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1.0  EXECUTIVE  SUMMARY 


We  compared  classification  capabilities  for  handheld  electromagnetic  sensors  that  were 
deployed  using  either  a  grid-based  template  for  spatial  registration  or  an  IMU-based 
positioning  system.  Demonstration  data  were  collected  at  the  Aberdeen  Proving  Ground 
using  both  systems.  The  two  data  sets  were  analyzed  separately  and  submitted  to  the 
Institute  for  Defense  Analyses  for  grading. 

The  IMU-based  positioning  system  involved  in  this  demonstration  is  known  as  the  Small 
Area  Inertial  Navigation  and  Tracking  (SAINT)  system.  It  was  developed  by  ENSCO  as 
part  of  an  earlier  ESTCP  project  and  enhanced  and  integrated  with  the  EM61HH  sensor  in 
support  of  this  effort.  The  SAINT  system  consists  of  a  Honeywell  HG1900  IMU  and  a 
LEICA  DMC.  The  Honeywell  HG1900  IMU  consists  of  orthogonally  aligned  MEMS 
accelerometers  and  gyroscopes  that  record  3-axis  acceleration  and  rotation  rates, 
respectively  enclosed  in  an  8-cubic  inch  container.  The  LEICA  DMC  is  employed  to  aid 
the  IMU  and  constrain  heading  drift.  The  digital  magnetic  compass  measures  the  strength 
and  direction  of  a  magnetic  field  and  can  be  used  to  determine  magnetic  north  in  an 
environment  free  of  additional  magnetic  fields.  SAINT  records  all  IMU  and  digital 
magnetic  compass  data  onto  an  internal  compact  flash  card. 

The  concept  of  operations  for  the  SAINT+EMI  sensing  system  consists  of  the  following 
steps;  initial  zero  velocity  update  (ZUPT),  acquire  data,  final  ZUPT.  During  the  initial 
ZUPT,  the  SAINT  unit  is  set  down  on  the  tripod  mount  and  remains  stationary  for  15 
seconds.  During  data  collection,  the  operator  picks  up  the  unit  and  acquires  geophysical 
data  for  30  seconds  or  less  of  free  navigation  over  the  area  of  interest.  The  data  collection 
event  ends  when  the  SAINT  unit  is  returned  to  its  original  location  on  the  tripod  mount  for 
about  15-seconds  final  ZUPT. 

The  template  consisted  of  a  6  by  6  square  grid  of  points  with  a  spacing  of  0.15  m.  Each 
grid  had  38  measurements  with  the  first  and  last  being  in  the  air  to  zero  the  sensor. 
Template  measurements  took  roughly  3-4  minutes  a  cell  with  a  comparable  average  daily 
rate  to  the  SAINT.  Logistically,  the  wooden  template  had  to  be  moved  cell  to  cell  which 
was  comparable  to  moving  the  SAINT  tripod  cell  to  cell.  Both  the  SAINT  and  template 
data  were  fit  with  the  same  EMI  model.  Discrimination  decisions  were  based  on  the  same 
library  fit  and  ratio  test. 

Overall,  excellent  response  stage  and  discrimination  stages  results  were  obtained  for  both 
positioning  systems.  For  the  response  stage,  the  EM61-HH  was  manually  swept  over  the 
center  of  each  cell.  If  there  was  no  measurable  signal  (~  10  mV)  within  roughly  0.25  m  of 
the  cell  center,  the  cell  was  determined  to  be  empty  for  both  data  sets  and  no  further 
measurements  were  taken.  The  EM61-HH  missed  only  105mm’s  (projectile  and  HEAT) 
and  only  at  their  deepest  depth  range  (8-12  times  their  diameter)  with  a  Pd  of  0.67  in  this 
range.  All  other  P/s  were  1.0  with  an  overall  P(/  of  0.99. 
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The  template  data  and  SAINT  data  were  processed  with  the  same  discrimination  algorithm 
with  the  same  threshold  settings.  The  template  data  achieved  an  overall  discrimination 
stage  Pd  of  0.93  compared  to  0.99  for  the  SAINT.  With  regard  to  efficiency,  which  is  a 
measure  of  how  much  of  the  ordnance  detected  was  identified  as  ordnance  in  the 
discrimination  stage,  the  SAINT  system  identified  all  detected  ordnance  as  being  ordnance 
and  had  a  score  of  1.0  while  the  template’s  score  was  0.93. 
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2.0  INTRODUCTION 


2.1  BACKGROUND 

The  billion-plus  estimated  cost  of  cleaning  up  land  contaminated  with  unexploded 
ordnance  (UXO)  and  related  materials  is  significantly  affected  by  the  costs  associated 
with  excavating  objects  that  pose  no  harm.  The  Defense  Science  Board  has  calculated 
that  if  the  false  alarm  rate  can  be  reduced  from  100:1  to  10:1,  the  economics  of  UXO 
remediation  can  be  inverted  from  75%  of  the  cost  devoted  to  digging  false  alarms  to  75% 
devoted  to  digging  UXO  [1], 

Differentiating  between  UXO  and  clutter  using  shape  information  derived  from 
electromagnetic  induction  (EMI)  data  has  proven  difficult  during  the  past  decade. 
Although  notable  success  has  been  realized  by  the  research  community  using  data 
acquired  in  very  controlled  settings,  errors  inherent  with  spatially  registering  field  data 
have  significantly  limited  discrimination  performances.  It  turns  out  that  position 
estimates  possessing  sub-centimeter  precision  are  required.  When  combined  with  data 
density  challenges,  signal-to-noise  issues,  and  non-unique  solutions,  these  issues  have 
dramatically  tempered  discrimination  performances.  The  impact  on  the  Department  of 
Defense  is  clear  -  and  expensive.  The  widespread  view  that  discrimination  is  not  possible 
drives  the  requirement  that  all  detected  objects  must  be  removed.  This  in  turn,  drives  the 
total  cleanup  cost,  which  is  estimated  in  the  billions  of  dollars.  This  project  was  directed 
towards  demonstrating  improved  discrimination  performances  in  field  settings  by 
precisely  spatially  registering  handheld  EMI  sensor  measurements  using  a  tactical  grade 
inertial  navigation  tracking  system. 

The  current  approach  to  mitigating  sensor  positioning  problems  associated  with  dynamic 
surveys  involves  collecting  data  on  a  fixed  grid  over  the  target  using  a  template  laid  on 
the  ground.  Although  inexpensive,  this  approach  leaves  much  to  be  desired.  A  more 
desirable  positioning  solution  is  one  that  is  integral  to  the  sensor,  allows  the  operator  to 
sweep  the  sensor  back  and  forth  with  ease,  and  permits  the  data  density  to  be  dynamically 
adjusted.  This  would  permit  use  in  rougher  terrain  and  confined  locations.  This  project 
integrates  a  tactical  grade  inertial  navigation  and  tracking  system  with  a  commercial 
EM61HH-MK2  sensor.  The  inertial  navigation  tracking  system  developed  by  ENSCO 
consists  of  a  Honeywell  HG1900  inertial  measurement  unit  (IMU)  and  a  LEICA  digital 
magnetic  compass  and  has  demonstrated  sub  centimeter  precision.  The  EM61HH-MK2 
is  a  hand-held  complement  to  the  EM61-MK2,  providing  greater  sensitivity  to  smaller 
targets  at  shallow  depths. 
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2.2  OBJECTIVE  OF  THE  DEMONSTRATION 


Our  objective  is  to  establish  classification  performance  for  a  handheld  data  collection 
system  that  combines  an  EMI  sensor  and  an  IMU. 

2.3  REGULATORY  DRIVERS 

Stakeholder  acceptance  of  the  use  of  discrimination  techniques  on  real  sites  will  require 
demonstration  that  these  techniques  can  be  deployed  efficiently  and  with  high  probability 
of  discrimination.  The  first  step  in  this  is  to  demonstrate  acceptable  performance  on  a 
test  site  such  as  that  at  Aberdeen.  After  that  hurdle  has  been  passed,  successful 
demonstration  at  a  live  site  will  facilitate  regulatory  acceptance  of  the  methods. 


4 


3.0  TECHNOLOGY 


3.1  TECHNOLOGY  DESCRIPTION 

The  principal  objective  of  this  demonstration  is  to  successfully  classify  obscured  UXO 
targets  by  inverting  geophysical  data  that  are  spatially  positioned  using  an  IMU,  which 
precisely  positions  geophysical  measurements,  relative  to  each  other,  as  they  acquired  as 
a  function  of  time.  To  accomplish  this  objective,  we  leveraged  ENSCO’s  Small  Area 
Inertial  Navigation  and  Tracking  (SAINT;  Figure  3-1).  The  photos  (from  left  to  right) 
show  a  SAINT-aided  EM61HH  being  used  on  a  wooden  platform  designed  for  routine 
testing.  The  concept  of  operation  (CONOPS)  includes  setting  the  sensing  system 
(EM6 1/SAINT  plus  tripod)  within  a  meter  or  two  of  an  anomaly  location.  The  SAINT  is 
rigidly  attached  to  the  EM61  handle.  The  combined  system  is  placed  on  a  tripod  and 
powered  up.  After  sitting  idle  for  15  seconds  of  background  data,  the  SAINT  beeps  once 
and  flashes  the  blue  LED.  The  user  lifts  the  system  and  proceeds  to  sweep  the  sensor 
back  and  forth  over  the  object  location.  The  SAINT  beeps  and  flashes  at  several  intervals 
during  data  collection.  After  30  seconds  of  collection  it  beeps  and  lights  up  continuously 
until  replaced  on  the  tripod.  After  another  1 5  seconds  of  being  idle,  it  beeps/flashes  once 
to  indicate  that  it  is  ready  to  move  again.  Examples  of  actual  sweep  data  are  shown  below 
the  photos  below.  After  the  idle  period,  more  data  can  be  collected  over  the  same  object 
or  from  nearby  objects  without  moving  the  tripod  if  desired. 


-500  ■■■■.■■■■■■■■■■■ 

0  50  100  150 

Sample 


Figure  3-1  Standard  survey  procedures  include  taking  the  sensor  off  of  the  tripod, 
waving  it  above  the  targets’  location,  and  returning  the  sensor  to  the  tripod  for  a  zero 
velocity  update  (top  photographs).  The  two  scatter  plots  on  the  bottom  show  the  spatial 
trajectory  of  the  sensor  head  (bottom  left)  and  the  measured  EMI  data  (bottom  right). 
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3.1.1  EMI  Sensor  -  EM61HH 


We  utilized  an  EM61HH  sensor  manufactured  by  Geonics.  The  EM61HH  is  a  hand-held 
complement  to  the  EM61,  providing  greater  sensitivity  to  smaller  targets  at  shallow 
depths.  Data  is  collected  from  an  air-core,  17cm  diameter,  single  receive  coil  at  four  time 
gates,  geometrically  spaced  in  time  from  147  ps  to  613  ps,  after  transmitter  turn-off. 
With  a  smaller  spatial  footprint  than  the  standard  EM61,  the  EM61HH  is  relatively  less 
sensitive  to  sources  of  potential  interference.  The  small  footprint  also  provides  enhanced 
target  resolution  and,  consequently,  improves  discrimination  of  multiple  targets. 

Effective  cued  identification  using  the  EM61HH  has  been  demonstrated  by  AETC  in 
ESTCP  project  200108  (Handheld  Sensor  for  Unexploded  Ordnance  Discrimination).  As 
reported  at  the  2002  SERDP/ESTCP  Partners  in  Environmental  Technology  Symposium, 
the  sensor  technology  was  demonstrated  at  a  Brownfield  Redevelopment  site  and 
achieved  excellent  results  in  discriminating  between  UXO  items  (37mm,  47mm  and 
75mm  projectiles)  and  various  industrial  and  cultural  clutter  items  [2],  For  that 
demonstration,  the  sensor  positioning  problem  was  solved  by  collecting  data  on  a  fixed 
grid  over  the  target  using  a  template  laid  on  the  ground  (Figure  3-2). 


Figure  3-2  A  standard  approach  to  mitigating  sensor  positioning  problem  involves 
collecting  data  on  a  fixed  grid  over  the  target  using  a  template  laid  on  the  ground.  The 
sensor  in  the  photograph  is  an  EM61HH-MK2. 

3.1.2  Data  Analysis 

The  basic  electromagnetic  induction  response  model  involves  a  simple  dipole  response 
[3].  Inversion  of  this  model  solves  for  the  object’s  location,  object  orientation,  and  the 
magnetic  polarization  response  terms.  The  polarization  terms,  referred  to  here  as  /?’s, 
determine  the  strength  of  the  object’s  induced  response  along  the  object’s  physical  axes. 
It  is  these  terms  that  allow  for  discrimination  between  buried  ordnance  and  other  metallic 
clutter.  For  elongated  ferrous  ordnance  items  such  as  a  projectile  or  mortar,  there  is  one 
large,  primary  /?  response  along  the  item’s  long  axis  and  two  smaller,  secondary 
responses  transverse  to  this  f3x>  J3y  -  f3z .  Accurately  determining  these  parameters 

from  model-based  inversion  is  limited  by  the  signal-to-noise  ratio  of  the  sensor  data  and 
by  the  positioning  accuracy  of  the  sensor  trajectory.  In  fact,  simulations  suggest  that 
positioning  accuracies  of  better  than  one  centimeter  are  required  to  invert  /?’  s  with  less 
than  20%  error. 
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When  sweeping  the  EM61HH  back  and  forth  over  a  target,  it  has  been  previously  noted 
[5]  that  the  peak  EM61HH  response  was  both  delayed  and  distorted  relative  to  the  actual 
target  location.  The  receiver  output  of  this  sensor  is  analog  integrated  with  a  filter  that 
both  shifts  and  distorts  the  sensor’s  response.  This  filter  can  sufficiently  distort  signal 
shape  to  limit  the  inversion  process.  This  impacts  the  accuracy  of  target  parameter 
estimates  from  dipole  inversion  at  sweeping  speeds  of  more  than  a  few  cm/sec.  At  such 
speeds,  it  is  necessary  to  account  for  the  sensor's  temporal  response.  Figure  3-3 
compares  inversion  results  with  the  static  (upper  plot)  and  dynamic  (lower  plot)  response 
models.  In  these  figures,  the  primary  (largest)  P  is  plotted  along  the  horizontal  axis,  and 
the  secondary  Ps  are  plotted  along  the  vertical  axis,  with  a  symbol  at  the  mean  value  and 
a  line  running  between  the  two.  For  the  test  objects  (two  spheres  and  two  projectiles),  the 
two  transverse  beta  values  are  expected  to  be  equal.  Furthermore,  for  the  two  spheres 
considered,  all  three  betas  (i.e.  longitudinal  and  transverse)  are  expected  to  be  equal.  The 
dotted  line  represents  the  line  where  longitudinal  and  transverse  values  are  equal.  The 
dynamic  model  generally  produces  tighter  eigenvalue  clusters  and  better  shape 
characterization  (secondary  Ps  more  nearly  equal)  than  does  the  static  model. 

Using  empirical  tests,  the  sensors’  temporal  response  was  determined  and  incorporated 
into  the  dipole  inversion  models  used  for  this  demonstration. 


Figure  3-3  Inversion  results  showing  polarizations  (P’s)  when  the  dynamic  response  is 
not  taken  into  account  (top)  versus  when  it  is  taken  into  account  (bottom). 

3.1.3  Spatial  Registration  System 

For  recording  the  sensors  3-D  position,  we  used  ENSCO’s  SAINT  system,  which  was 
originally  developed  in  2007  [4]  and  enhanced  during  this  program.  It  is  a  stand-alone 
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unit  (Figure  3-4).  The  inertial  navigation  tracking  system  consists  of  a  Honeywell 
HG1900  inertial  measurement  unit  (IMU)  and  a  LEICA  digital  magnetic  compass 
(DMC).  The  Honeywell  HG1900  IMU  consists  of  orthogonally  aligned  micro  electro¬ 
mechanical  systems  (MEMS)  accelerometers  and  gyroscopes  that  record  3 -axis 
acceleration  and  rotation  rates,  respectively  enclosed  in  an  8-cubic  inch  container.  The 
LEICA  DMC  is  employed  to  aid  the  IMU  and  constrain  heading  drift.  The  digital 
magnetic  compass  measures  the  strength  and  direction  of  a  magnetic  field  and  can  be 
used  to  determine  magnetic  north  in  an  environment  free  of  additional  magnetic  fields. 
With  the  exception  of  batteries,  a  tripod  stand,  and  the  post-processing  personal  computer 
(PC),  all  components  of  SAINT  are  packaged  into  a  single  enclosure.  The  prototype 
SAINT  system  [4,  5]  achieved  spatial  registration  accuracies  of  <lcm. 


Figure  3-4  Left:  SAINT  attached  to  the  handle  of  an  EM61HH-MK2.  Right:  Honeywell 
HG1900  IMU  and  LEICA  DMC,  which  are  housed  within  the  yellow  enclosure. 

As  part  of  this  project,  we  built  an  improved  SAESIT  sensor  and  integrated  it  fully  with 
the  EM61HH  sensor.  The  changes  made  significant  structural  changes  to  SAINT  in  order 
to  make  the  system  more  efficient  and  fieldable  while  still  maintaining  the  original 
systems  accuracy  and  ease  of  use  (Figure  3-5  and  Figure  3-6).  The  improvements 
included  (i)  mitigating  synchronization  timing  issues  between  the  SAINT  and  EM61HH; 
(ii)  migrating  the  data  acquisition  software  from  an  embedded  processor  to  a  single  board 
computer  (SBC);  (iii)  redesigning  the  hardware  for  easier  access;  (iv)  fabricating  a  non- 
metallic  docking  station;  and  (v)  improving  the  calibration  procedures.  See  Appendix  A 
for  details  regarding  the  system  improvements  and  validation  tests. 
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Figure  3-5  A  comparison  of  the  actual  (2-D  motion,  pen  trace  on  paper)  versus  the 
computed  position  from  the  SAINT  system.  Top  right:  The  black  line  represents  the 
actual  path,  and  the  red  line  represents  the  path  estimated  using  the  R-T-S  Smoother.  The 
blue  ellipses  correlate  to  the  95%  confidence  errors  estimated  by  post-processing 
software. 


Figure  3-6  The  battery  (left)  and  tripod  (right)  were  redesigned  during  this  project. 
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3.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  SAINT  positioning  system  was  integrated  with  the  EM61HH  sensor  and  records 
position  data  with  the  sensor  data.  The  advantage  of  the  SAINT  compared  to  the 
traditional  grid  template  method  is  the  ability  to  sweep  the  sensor  back  and  forth  with 
ease  and  dense  and  varied  data  collections  and  sweep  patterns.  This  is  especially 
valuable  for  large  footprint  anomalies  because  a  larger  spatial  grid  (out  to  the 
background)  can  be  collected  whereas  this  is  not  feasible  or  efficient  using  a  fixed 
template.  Similarly,  for  smaller  targets  the  sensor  can  be  moved  slower  over  a  smaller 
area  to  increase  the  data  density  over  the  target. 

The  SAINT  would  permit  use  in  rougher  terrain  and  confined  locations  that  may  not  be 
conducive  to  placing  a  rigid  template  on  the  ground.  Many  times  vegetation  or  other 
natural  obstructions,  such  as  rocks  and  stumps,  do  not  allow  a  grid  template  to  be  placed 
securely  on  the  ground.  The  ability  of  the  SAINT  system  to  sweep  around  these 
obstructions  is  significant  advantage  when  surveying  in  a  treed  environment. 

Another  advantage  of  the  SAINT  system  is  data  production  rate.  It  is  anticipated  that  an 
operator  can  interrogate  twice  as  many  targets  compared  to  the  grid  template  method 
during  a  fixed  period  of  time.  The  position  data  will  also  be  more  reliable  as  operator 
fatigue  and  error  are  more  pronounced  using  the  grid  template.  Over  the  course  of  the 
day  and  project,  sensor  operators  tend  to  be  less  careful  when  placing  the  sensor  on  the 
grid  template.  Also,  the  operator  may  accidentally  skip  or  add  grid  nodes  when  using  the 
grid  template.  Most  of  the  time  it  is  impossible  to  tell  with  certainty  which  grid  nodes 
were  skipped  during  collection  so  the  data  is  discarded  and  the  target  is  resurveyed. 
Using  the  SAINT  system  should  alleviate  these  problems. 

The  main  limitation  of  the  SAINT  system  is  the  duration  of  free  navigation  time  before 
the  system  must  be  set  down  and  a  zero  velocity  update  (ZUPT)  performed.  Longer  data 
collection  times  would  facilitate  use  and  decrease  total  operation  time.  IMU  position 
errors  grow  quadratically  with  the  duration  of  free  navigation  between  ZUPTs.  For  a 
given  IMU,  shorter  free  navigation  leads  to  smaller  maximum  errors,  longer  free 
navigation  leads  to  larger  free  navigation  errors.  A  better  quality  IMU  would  extend  the 
useable  free  navigation  time  and/or  reduce  the  maximum  error.  We  selected  the  HG1900 
as  a  tradeoff  between  price,  size/weight,  and  performance  of  available  IMUs. 

This  demonstration  used  an  EM61HH  sensor  with  the  hand-held  coils  which  is  a  standard 
commercial  UXO  sensor.  The  size  of  coils  makes  it  easy  to  move  the  sensor  and  collect 
cued  data.  Unfortunately,  the  transmit  power  and  smaller  coil  size  also  limit  the  sensors 
ability  to  detect  most  ordnance  at  their  maximum  burial  depth. 

A  known  limitation  to  the  data  analysis  results  from  non-unique  inversion  results.  Using 
EMI  data,  an  estimate  of  the  objects’  shape  can  be  made.  Specifically,  the  eigenvalues  of 
the  magnetic  polarizability  tensor  provide  the  shape  information.  There  are  three 
eigenvalues  -  each  corresponds  to  a  principal  axis  of  object.  Discrimination  is  possible 
only  to  the  degree  that  the  eigenvalues  are  different.  In  other  words,  even  with  ideal  data, 
the  estimated  burial  depth,  apparent  size,  and  shape  features  may  not  separate  UXO  and 
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clutter  signatures  into  distinct,  non-overlapping  classes.  This  is  because  the  anomaly 
features  derived  from  EMI  and  magnetic  data  are  not  necessarily  unique  to  UXO.  Clutter 
items  that  have  similar  shapes  and  burial  attributes  to  ordnance  can  have  geophysical 
signatures  that  are  indistinguishable  from  UXO  signatures  and,  as  such,  will  have  similar 
eigenvalues  and  therefore  likely  be  classified  as  ordnance.  Examples  include  items  such 
as  pipes,  post  sections  and  axial  symmetrical  fragments. 
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4.0  PERFORMANCE  OBJECTIVES 


The  performance  objectives  for  this  demonstration  are  summarized  in  Table  4-1. 
Table  4-1.  Performance  Objectives  for  This  Demonstration 


Performance 

Objective 

Metric 

Data  Required 

Success 

Criteria 

Performance 

Quantitative  Performance  Objectives 

Maximize 

correct 

classification  of 
UXO 

Number  of  UXO 
retained 

•  Ranked 
Anomaly  list 

•  Scoring  report 

Approach 
correctly 
classifies 
>95%  of  UXO 

Yes 

Maximize 

correct 

classification  of 
clutter 

Number  of  false 
alarms 
eliminated  at 
demonstrator 
operating  point. 

•  Ranked 
anomaly  lists 

•  Scoring  report 

Reduction  of 
false  alarms  by 
>  40%  while 
retaining 
>95%  of 
detected 
munitions 

Yes 

Production  Rate 

Number  of 
targets 
interrogated 
each  day 

•  Log  of  field 
work 

2x  the  number 
of  targets  using 
grid  template 

No 

Qualitative  Performance  Objective 

Ease  of  Use 

•  Feedback 
from  operator 
on  ease  of  use 

Yes 

4.1  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  UXO 

This  is  one  of  the  two  primary  measures  of  the  effectiveness  of  the  classification 
approach.  By  collecting  high-quality  data  and  analyzing  those  data  with  advanced 
parameter  estimation  and  classification  algorithms  we  expect  to  be  able  to  classify  the 
targets  with  high  efficiency. 

4.1.1  Metric 

The  metric  is  the  number  of  UXO  items  that  were  correctly  classified  as  UXO 

4.1.2  Data  Requirements 

Data  requirements  include  ground  truth  information  and  inverted  features. 
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4.1.3  Success  Criteria 


The  objective  was  considered  met  if  <95%  of  the  UXO  are  correctly  labeled  as  UXO. 

4.2  OBJECTIVE:  MAXIMIZE  CORRECT  CLASSIFICATION  OF  CLUTTER 

This  is  a  primary  measure  of  the  effectiveness.  By  collecting  high-quality,  precisely- 
located  data,  we  were  able  to  discriminate  detectable  munitions  from  scrap  and  frag  with 
high  efficiency. 

4.2.1  Metric 

We  compared  the  number  of  false  targets  that  can  be  eliminated  using  the  demonstrated 
discrimination  procedures  for  both  the  SAINT  and  grid  template  positioned  data  to  the 
total  number  of  false  targets.  We  prepared  a  ranked  dig  list  for  the  targets  we  interrogate 
with  a  dig/no-dig  threshold  indicated  and  ATC  personnel  used  their  automated  scoring 
algorithms  to  assess  our  results. 

4.2.2  Data  Requirements 

The  identification  of  the  items  in  the  test  field  is  known  to  the  test  site  operators.  Our 
ranked  dig  list  is  the  input  for  this  standard  and  ATC’s  standard  scoring  is  the  output. 

4.2.3  Success  Criteria 

The  objective  was  considered  met  if  the  SAINT  positioned  data  performed  equal  to  or 
better  than  the  grid  template  data.  Additionally,  for  the  anomalies  that  have  a  SNR 
sufficient  for  analysis,  eliminating  more  than  40%  of  the  non-munitions  items  while 
retaining  95%  of  the  munitions  items  on  the  dig  list  was  deemed  a  success. 

4.3  OBJECTIVE:  PRODUCTION  RATE 

One  of  the  objectives  of  using  the  SAINT  to  position  the  sensor  is  to  increase  the  number 
of  targets  that  are  investigated  each  day.  The  increase  in  production  should  outweigh  the 
increased  cost  of  the  equipment. 

4.3.1  Metric 

The  number  of  targets  interrogated  per  day  is  the  metric  for  this  objective.  We  compared 
the  number  using  the  SAINT  positioned  system  to  the  traditional  grid  template. 

4.3.2  Data  Requirements 

Survey  productivity  will  be  determined  from  a  review  of  the  demonstration  field  logs. 

4.3.3  Success  Criteria 

The  objective  was  considered  met  if  at  least  twice  as  many  targets  are  interrogated  each 
survey  day  using  the  SAINT  positioned  system. 
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4.4  OBJECTIVE:  EASE  OF  USE 


This  qualitative  objective  was  intended  as  a  measure  of  the  long-term  usability  of  the 
technology. 

4.4.1  Data  Requirements 

This  objective  was  evaluated  based  on  operator  feedback 
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5.0  SITE  DESCRIPTION 


5.1  SITE  SELECTION 

This  demonstration  was  conducted  at  the  APG  Standardized  UXO  Technology 
Demonstration  Site  located  at  the  Aberdeen  Proving  Ground,  MD.  Use  of  this  site 
allowed  us  to  receive  validation  results  from  near-real-world  conditions  without  incurring 
the  logistics  and  intrusive  investigation  expenses  that  is  required  for  a  demonstration  at  a 
live  site. 

5.2  SITE  HISTORY 

The  APG  Standardized  Test  Site  is  located  within  a  secured  range  area  of  the  Aberdeen 
Proving  Grounds.  The  Aberdeen  Area  of  APG  is  located  approximately  30  miles 
northeast  of  Baltimore  at  the  northern  end  of  the  Chesapeake  Bay.  The  Standardized 
UXO  Technology  Demonstration  Site  is  adjacent  to  the  Trench  Warfare  facility  at  the 
Aberdeen  Proving  Ground.  The  specific  area  was  used  for  a  variety  of  ordnance  tests 
over  the  years.  Initial  magnetometer  and  EMI  surveys  conducted  by  the  MTADS  team 
performed  after  a  “mag  and  flag”  survey  of  the  same  area  identified  over  a  thousand 
remaining  anomalies.  These  data  were  used  for  a  final  clean  up  of  the  site  prior  to  the 
emplacement  of  the  original  test  items.  Prior  to  the  two  subsequent  reconfiguration 
events,  unexplained  anomalies  identified  by  demonstrators  using  the  site  were  also 
investigated  and  removed. 

5.3  SITE  TOPOGRAPHY  AND  GEOLOGY 

The  demonstration  site  is  relatively  flat  and  conducive  to  acquiring  high  quality 
geophysical  data.  The  local  near  surface  materials  consist  of  very  deep,  slowly 
permeable,  poorly  drained  soils  [6].  Slopes  range  from  0  to  2  percent. 


5.4  MUNITIONS  CONTAMINATION 

The  area  currently  occupied  by  the  Standardized  site  has  seen  an  extensive  history  of 
munitions  use.  Historical  records  provided  by  ATC  and  previous  remediation  results 
indicated  that  the  likely  munitions  of  interest  for  this  site  are: 

•  Grenades,  Mkl,  Mkll,  and  French  VB  Rifle  without  chute 

•  Grenades,  French  VB  Rifle  w/  chute 

•  60mm  mortars  (including  2”  Smoke) 

•  3”  Stokes  (Smoke  and  HE) 

•  105  mm  projectiles 

•  155  mm  proj  ectiles 
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5.5  SITE  GEODETIC  CONTROL  INFORMATION 


There  are  two  first-order  survey  points  on  the  site  for  use  as  GPS  base  station  points  (see 
Table  5-1  for  details).  The  horizontal  datum  for  all  values  is  NAD83.  The  vertical 
control  is  referenced  to  the  NAVD88  datum  and  the  Geoid03  geoid. 

Table  5-1:  Geodetic  control  at  the  APG  Demonstration  Site 


ID 

Latitude 

Longitude 

Elevation 

Northing 

Easting 

HAE 

477 

39°  28’  18.63880”  N 

76°  07’  47.71815”W 

10.669  m 

4,369,749.013 

402,810.038 

-22.545 

478 

39°  28’  04.24219”  N 

76°  07’  48.50439”W 

11.747  m 

4,369,305.416 

402,785.686 

-21.473 

5.6  SITE  CONFIGURATION 

Figure  5-1  is  a  map  of  the  Standardized  UXO  Technology  Demonstration  Site  at  APG. 
The  Calibration  and  Blind  Grids  are  shown  along  with  the  various  Open  Field  Areas. 


Figure  5-1  Map  of  the  APG  Standardized  UXO  Test  Site. 
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6.0  TEST  DESIGN 


6.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  demonstration  was  executed  in  two  stages.  The  first  stage  included  collecting  cued 
EM61HH  data  using  a  grid  template  while  the  second  stage  used  the  SAINT  to  position 
the  sensor  head. 

The  data  from  both  the  template  positioned  and  SAINT  positioned  surveys  were  inverted 
using  the  data  analysis  methodology  discussed  in  Section  7.  Estimated  target  parameters 
as  well  as  the  classification  decisions  were  submitted  to  AEC  for  scoring. 

6.2  SITE  PREPARATION 

The  Standardized  UXO  Test  Sites  are  configured  with  clearly-marked  calibration,  blind 
validation,  and  open  field  scenarios.  Two  GPS  control  points  are  also  provided.  Basic 
facilities  such  as  portable  toilets  and  field  buildings  are  also  provided.  No  additional  or 
special  site  preparation  was  required  for  this  demonstration. 

6.3  SYSTEMS  SPECIFICATION 

6.3.1  Time-domain  EMI  Sensor 

The  EM61HH  was  used  for  this  demonstration.  We  configured  the  sensor  to  record  the 
four  standard  factory-programmed  time  gates  (0.147ms,  0.263ms,  0.414ms  and  0.613ms). 

6.3.2  Grid  Template 

The  EMI  data  were  collected  on  a  6x6  point  square  grid  (nodes  separated  by  15cm 
resulting  in  a  75x75cm  area).  The  grid  template  was  placed  directly  on  the  ground,  with 
the  EM61HH  positioned  directly  on  the  grid  (Figure  6-1).  Sensor  location  on  the  grid 
was  controlled  by  attaching  a  plexiglass  guide  with  crosshairs  to  the  bottom  of  the  sensor 
coils  and  lining  up  the  cross  hairs  with  grid  line  intersections.  EM61HH  data  were 
recorded  using  the  handheld  field  computer  and  software  that  is  part  of  the  standard 
equipment  package.  The  software  was  set  to  record  all  data  for  each  target  in  one  survey 
line  with  the  data  file  consisting  of  multiple  survey  lines.  The  data  were  recorded 
continuously  for  each  target  at  10  readings/second  and  the  fiducial  marker  used  to  flag 
the  times  when  the  sensor  is  at  rest  over  each  grid  node. 
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Figure  6-1:  Template  used  to  collect  EM61HH  array  data  during  reacquisition. 

6.3.3  SAINT 

The  SAINT  system  consists  of  a  Honeywell  HG1900  IMU  and  a  LEICA  DMC.  The 
Honeywell  HG1900  IMU  consists  of  orthogonally  aligned  MEMS  accelerometers  and 
gyroscopes  that  record  3-axis  acceleration  and  rotation  rates,  respectively  enclosed  in  an 
8-cubic  inch  container.  The  LEICA  DMC  is  employed  to  aid  the  IMU  and  constrain 
heading  drift.  The  digital  magnetic  compass  measures  the  strength  and  direction  of  a 
magnetic  field  and  can  be  used  to  determine  magnetic  north  in  an  environment  free  of 
additional  magnetic  fields.  SAINT  records  all  IMU  and  digital  magnetic  compass  data 
onto  an  internal  compact  flash  card  (CF). 

Geophysical  data  were  acquired  in  <30-second  cycles  after  which  the  IMU  was  returned 
to  its  initial  position.  Each  data  collection  event  consists  of  the  following  steps;  initial 
ZUPT,  acquire  data,  final  ZUPT.  During  the  initial  ZUPT,  the  SAINT  unit  is  set  down 
on  the  tripod  mount  and  remains  stationary  for  15  seconds.  During  data  collection,  the 
operator  picks  up  the  unit  and  acquires  geophysical  data  for  30  seconds  or  less  of  free 
navigation  over  the  area  of  interest.  The  data  collection  event  ends  when  the  SAINT  unit 
is  returned  to  its  original  location  on  the  tripod  mount  for  about  15 -seconds  final  ZUPT. 

Lights  and  audible  signals  aid  the  user  for  proper  operation.  During  the  30  second  free 
navigation  period,  various  audible  chirps  are  sounded  with  10,  5,  and  0  seconds 
remaining  before  a  new  ZUPT  is  required.  Data  density  can  be  increased  by  resurveying 
any  given  anomaly. 


6.4  CALIBRATION  ACTIVITIES 

Spatial  measurements  were  made  to  record  the  three-dimensional  position  offset  of  the 
center  of  sensitivity  of  the  EM61HH,  which  is  defined  to  be  the  physical  center  of  mass 
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of  the  figure-eight  coil  configuration,  with  respect  to  the  IMU  in  the  measurement  frame 
of  the  IMU. 

Daily  calibration  efforts  consisted  of  collecting  cued  EM61HH  data  over  a  standard 
object.  The  data  were  collected  using  the  same  procedures  as  used  over  training  and 
blind  targets.  The  daily  variations  in  the  system  response  were  monitored  and  remained 
within  10%  of  the  reference  values. 

6.5  DATA  COLLECTION 
6.5.1  Scale  of  the  Demonstration 

The  demonstration  was  conducted  at  the  calibration  and  blind  grids  of  the  APG 
Standardized  UXO  Test  Site. 

6.6  SAMPLE  DENSITY 

EM61HH  data  spacing  using  the  grid  template  is  fixed  at  15  cm  in  both  directions.  The 
choice  of  grid  spacing  was  based  on  results  of  simulations  of  data  inversion  sensitivity  as 
well  as  past  experience  and  was  concluded  to  be  a  good  compromise  of  sample  density 
and  data  collection  time. 

The  EM61HH  data  spacing  using  the  SAINT  depends  on  the  speed  the  operator  moves 
the  sensor  over  the  target.  Since  sensor  data  is  collected  at  discrete  time  intervals,  desired 
sensor  data  density  along  the  path  over  which  the  sensor  is  swept  can  be  obtained  by 
regulating  the  speed  at  which  the  sensor  is  swept.  Free  navigation  is  limited  to  30 
seconds  to  minimize  positioning  error.  To  cover  a  larger  area  than  can  be  swept  in  30 
seconds,  two  or  more  data  sweeps  must  be  performed  in  series  (separated  by  a  ZUPT).  A 
sample  sweep  pattern  is  shown  in  Figure  6-2.  Since  the  ZUPT  locations  are  the  same,  no 
special  operation  has  to  be  performed  in  the  post-processing  software  to  overlay  the 
adjacent  collections.  Anomalies  that  cover  a  large  spatial  area  will  typically  have  a  lower 
data  density  than  anomalies  with  a  small  spatial  area. 


Figure  6-2:  Sample  sweep  pattern  to  cover  a  2x2m  square  area  in  two  passes. 


19 


6.6.1  Quality  Checks 

For  the  grid  template  data,  all  QC  checks  and  processing  was  done  using  IDL  routines 
that  were  developed  and  refined  during  past  surveys.  The  initial  QC  checks  consisted  of 
reading  the  data  files,  splitting  them  into  grids,  verifying  that  each  grid  has  the  correct 
number  of  marked  data  segments  (36  grid  points  plus  starting  and  ending  background), 
and  making  raster  plots  of  the  data.  Additional  QC  checks  and  processing  will  be  done 
using  routines  that  extract  the  background  and  grid  point  readings  identified  by  event 
marks,  allow  display  and  editing  of  the  data  and  characterize  the  anomaly.  The 
anomalies  were  characterized  by  inverting  the  data  to  a  dipole  model  each  day,  in  order  to 
monitor  the  fit  quality,  which  is  a  measure  of  how  accurately  the  modeled  data  matches 
the  measured  data.  We  have  seen  in  past  surveys  that  the  fit  quality  decreases  when 
measurement  errors  increase.  The  most  common  measurement  error  is  positional  error 
caused  by  the  field  crew  not  ensuring  the  crosshairs  on  the  guide  are  aligned  with  the 
gridlines  on  the  template. 

Similar  to  the  grid  template  data,  the  SAINT  data  were  checked  by  monitoring  the  fit 
quality  output  from  the  inversion  algorithms.  The  fit  quality  decreases  with  both  low 
SNR  signals  and  with  poor  positioning. 

Any  data  set  which  was  deemed  unsatisfactory  by  the  data  analyst  was  flagged  and  not 
processed  further.  The  anomaly  corresponding  to  the  flagged  data  was  logged  for  future 
re-acquisition. 

6.6.2  Data  Handling 

EM61HH  data  using  the  grid  template  are  stored  electronically  as  collected  on  the 
Allegro  data  acquisition  computer  and  downloaded  via  serial  port  to  a  notebook  computer 
for  QC/analysis  using  Geonics  supplied  software.  EM61HH  data  positioned  using  the 
SAINT  are  also  stored  electronically  as  collected  on  the  SAINT  acquisition  computer  and 
downloaded  to  a  notebook  computer  using  a  USB  or  Ethernet  cable.  The  SAINT 
acquisition  computer  is  treated  as  another  drive  when  connected  to  another  computer  and 
data  is  transferred  using  standard  drag  and  drop  methods. 

Raw  data  and  analysis  results  were  backed  up  from  the  data  analyst’s  computer  to  flash 
cards  daily.  These  results  were  archived  on  an  internal  file  server  at  SAIC  at  the  end  of 
the  survey. 

6.7  VALIDATION 

With  the  exception  of  the  Calibration  Grid,  the  ground  truth  for  the  Standardized  sites  is 
held  back  from  individual  technology  demonstrators  to  preserve  the  utility  of  the  Blind 
Grid  and  Open  Field  Areas.  Analysis  results  (i.e.,  prioritized  dig  lists)  from  the  Blind 
Grid  were  submitted  to  ATC  and  the  program  office  for  performance  evaluation. 
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7.0  DATA  ANALYSIS  AND  PRODUCTS 


7.1  PREPROCESSING 

The  data  files  for  the  grid  template  data  nominally  contain  data  for  one  or  more  targets. 
Different  targets  are  identified  by  the  survey  line  numbers  in  the  file.  In  the  data  section 
following  the  survey  line  header,  column  1  is  an  event  marker  which  identifies  readings 
at  the  grid  locations  over  the  target.  The  symbol  “P”  is  the  event  marker  when  the  sensor 
coil  is  over  a  grid  location.  The  symbol  “E”  is  the  event  marker  when  the  sensor  coil  is 
being  moved  to  the  next  grid  location.  Each  survey  line  should  have  38  data  sequences 
with  event  marks.  The  first  and  last  sequences  are  background  readings  taken  in  the  air 
out  of  ground  effects.  The  middle  36  correspond  to  locations  on  the  36  point 
measurement  grid.  Each  of  the  38  data  sequences  contain  multiple  readings  when  the 
sensor  is  stationary.  The  stationary  readings  are  averaged  and  assigned  a  local  XY 
location  based  on  its  fixed  position  on  the  grid  template.  The  data  for  each  6x6  point  data 
grid  are  then  leveled  using  the  before  and  after  backgrounds. 

The  preprocessing  for  the  SAINT  exploits  the  operational  requirement  that  the  start  and 
stop  locations  of  the  SAINT  hardware  be  identical.  The  operator  can  free  navigate  for  up 
to  30  seconds,  at  which  time  the  unit  must  be  returned  to  the  same  place  it  started.  A 
tripod  is  used  to  simplify  the  return  of  the  hardware  to  the  identical  location.  Upon 
completion  of  the  ZUPT,  a  blue  indicator  light  illuminates  on  the  SAINT  enclosure 
signifying  the  operator  can  free-navigate  when  ready. 

The  post-processing  software  includes  a  GUI  that  requires  the  user  to  select  the  filenames 
for  processing  and  the  periods  to  process.  The  processing  consists  of  the  following 
components: 

1  A  pre-filter  for  detection  of  ZUPT  intervals 

2  Navigation  equations  and  a  Kalman  filter  [8] 

3  Rauch-Tung-Streibel  (R-T-S)  Smoother 

4  A  component  to  translate  the  IMU  position  &  attitude  to  the  geophysical  sensor 
based  on  the  static  3-D  position  &  orientation  offset  vectors 

5  A  component  to  interpolate  the  sensor  position  &  attitude  (recorded  at  up  to  600 
Hz)  to  the  recorded  EM61  times  (recorded  at  approximately  15  Hz) 

The  error-state  Kalman  filter  feeds  corrections  back  into  the  navigation  equations  for  both 
navigation  errors  and  IMU  sensor  errors  to  optimally  estimate  the  position  and  orientation 
of  the  IMU.  The  accuracy  of  these  corrections  is  primarily  a  function  of  the  quality  of  the 
IMU,  the  quality  of  the  aiding  measurements  to  the  IMU  and  the  quality  of  the  model 
defining  the  IMU  in  the  Kalman  filter.  The  accuracy  of  the  reported  errors  is  primarily  a 
function  of  the  quality  of  the  model  defining  the  IMU.  Since  the  IMU  contains  six 
sensors,  and  each  has  errors  from  bias,  drift,  alignment,  scale  factor,  nonlinearity, 
asymmetry,  and  other  sources,  more  than  50  states  in  the  Kalman  Filter  were  necessary  to 
optimally  model  the  IMU.  The  Kalman  filter  provides  an  optimal  estimate  of  position 
based  on  all  measurements  available  until  time  tk.  However,  additional  information 
contained  in  the  measurements  after  tk  can  further  improve  the  estimate  of  position.  An 
R-T-S  Smoother  [9]  was  implemented  to  compute  an  optimal  smoothed  path  estimate 
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utilizing  all  available  measurements  within  the  data  set. 

In  addition  to  the  position  and  attitude  estimates,  the  error  estimates  are  provided  for  each 
of  the  components  of  position  and  attitude.  Processing  software  outputs  positions  in  a 
relative  coordinate  system  with  the  origin  at  the  ZUPT  location  and  the  orientation 
constrained  by  the  compass. 

7.2  TARGET  SELECTION  FOR  DETECTION 

All  targets  within  the  calibration  and  blind  grids  were  surveyed. 

7.3  PARAMETER  ESTIMATION 

The  EM61HH  data  are  inverted  using  the  standard  induced  dipole  response  model 
wherein  the  effect  of  eddy  currents  set  up  in  the  target  by  the  primary  field  is  represented 
by  a  set  of  three  orthogonal  magnetic  dipoles  at  the  target  location  [10].  The  measured 
signal  is  a  linear  function  of  the  induced  dipole  moment  m,  which  can  be  expressed  in 
terms  of  a  time  dependent  polarizability  tensor  B  as 

m  =  UBUT  H0 

where  U  is  the  transformation  matrix  between  the  physical  coordinate  directions  and  the 
principal  axes  of  the  target  and  Ho  is  the  primary  field  strength  at  the  target.  The 
eigenvalues  (3i(t)  of  the  polarizability  tensor  are  the  principal  axis  polarizabilities. 

Given  a  set  of  measurements  of  the  target  response  with  varying  geometries  or  "look 
angles"  at  the  target,  the  data  can  be  inverted  to  determine  the  (X,  Y,  Z)  location  of  the 
target,  the  orientation  of  its  principal  axes  (\\ >,  0,  4>),  and  the  principal  axis  polarizabilities 
(Pi,  P2,  P3).  The  basic  idea  is  to  search  out  the  set  of  nine  parameters  (X,  Y,  Z,  \\i,  0,  (|), 
Pi,  p2,  P3)  that  minimizes  the  difference  between  the  measured  responses  and  those 
calculated  using  the  dipole  response  model. 

In  some  situations  (depending  on  target  size,  orientation  and  distance  from  the  sensor 
head)  there  is  a  bit  of  ambiguity  regarding  the  “correct”  values  of  the  P’s  and  the  depth. 
We  suspect  that  this  is  due  to  failure  of  the  dipole  response  model  to  faithfully  reproduce 
the  signal  when  the  dimensions  of  the  target  are  comparable  to  distances  over  which  there 
are  significant  changes  in  the  primary  field  (and  the  reciprocal  received  field).  For  most 
UXO,  there  is  one  large  P  corresponding  to  the  axial  response  and  two  smaller,  equal  P’s 
corresponding  to  transverse  responses.  With  ordnance  items  on  a  test  stand,  we  find  that 
the  depth  at  which  the  secondary  P’s  are  equal  is  not  always  the  depth  that  minimizes  the 
RMS  deviation  between  the  data  and  the  dipole  model.  Consequently  for  the  grid 
template,  the  EM  data  were  processed  by  “focusing”  the  measurement  array  at  depths 
from  ~5  cm  to  100  cm  below  the  ground  level.  As  a  function  of  focus  depth,  we  fit  the 
array  data  to  a  dipole  response  model  using  a  least  squares  procedure.  We  then  looked  at 
the  best  fit  eigenvalues  and  residual  error  as  functions  of  depth  to  find  the  “correct” 
values  of  the  P’s  and  the  depth. 
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There  are  several  additional  complications  in  inverting  the  EM61HH/SAINT  data.  The 
first  is  the  added  requirement  for  modeling  the  sensor’s  dynamic  time  response.  The 
receiver  output  of  this  sensor  is  analog  integrated  with  a  filter  that  both  shifts  and  distorts 
the  sensor’s  response.  This  filter  was  added  to  the  forward  model  used  by  the  inversion 
algorithm.  The  second  complication  is  the  sensor  bias  drift.  Over  minutes  of  data 
collection,  the  zero  level  of  the  sensor  changes.  The  data  collected  here  starts  with  the 
sensor  on  a  metal  tripod  with  an  unknown  offset.  It  then  moves  back  and  forth  over  an 
object  perhaps  reaching  zero  at  points,  perhaps  not.  Because  of  this,  an  offset  parameter 
has  been  added  to  the  data  inversion  process. 

7.4  CLASSIFIER  AND  TRAINING 

As  part  of  this  demonstration  and  under  previous  project  we  have  collected  in-air  data  for 
many  of  the  standard  APG  ordnance  targets.  We  also  collected  additional  in-air  training 
data  for  any  ordnance  not  already  in  our  library.  The  combination  of  these  data  was  used 
for  the  fit  library  entries.  Many  of  the  targets  are  composites  of  two  or  more  distinct 
parts,  like  a  steel  body  combined  with  an  aluminum  tail  assembly.  Depending  on  the 
distance  between  the  sensors  and  the  target,  such  items  can  exhibit  a  range  of  slightly 
different  EMI  signatures  corresponding  to  excitation  from  different  directions.  We 
included  measurements  with  the  target  oriented  nose  up,  nose  down,  flat  and  obliquely 
relative  to  the  sensor. 

Our  experience  at  our  Blossom  Point  test  site  has  been  that  polarizabilities  determined 
from  in-air  measurements  are  indistinguishable  from  those  determined  from 
measurements  taken  over  buried  targets.  We  used  data  from  the  calibration  lanes,  which 
contain  several  instances  of  each  target,  to  establish  that  this  holds  true  at  APG. 

We  used  target  features  for  clutter  items  found  in  the  calibration  grid.  Unfortunately 
there  were  only  eight  clutter  items  in  the  calibration  grid  so  we  augmented  our  clutter 
library  with  features  derived  from  previous  surveys  over  APG.  In  particular,  we  used  the 
results  from  the  ESTCP  project  200108  which  demonstrated  the  discrimination 
capabilities  of  the  EM61HH-MK2  over  the  blind  grid. 

Target  classification  was  based  on  a  library  matching  procedure.  We  compare  the  quality 
of  an  unconstrained  dipole  inversion  of  the  EM61HH-MK2  data  to  the  quality  of  a  dipole 
fit  constrained  by  principal  axis  polarizabilities  drawn  from  the  signature  library.  The 
library  values  were  based  on  the  mean  of  the  log  of  the  best  unconstrained  fits  from  the 
training  data.  Fit  quality  is  the  squared  correlation  coefficient  between  the  model  fit  and 
the  data.  If  the  ratio  of  the  constrained  fit  quality  to  the  unconstrained  fit  quality  (p)  is 
one,  then  the  library  item  is  as  good  a  match  to  the  data  as  possible.  If  the  ratio  is  small, 
then  the  library  item  is  a  poor  match. 

Rule  based  decisions  that  incorporate  the  fit  ratio,  polarizability  coefficients,  signal 
amplitude  and  fit  error  were  used  to  make  the  final  classification  decision.  The 
thresholds  were  decided  by  inspection  of  the  above  parameters  calculated  from  the 
training  data. 
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7.5  DATAPRODUCTS 


We  utilized  the  standard  reporting  templates  for  the  Blind  Grid  (Figure  7-1). 


BLIND  TEST  GRID 

Letter 

Number 

Response 

Stage 

Discrimination 

Stage/Ranking 

Classification 

(Use  B  for 

Blank) 

TYPE 

Depth  (M) 

Azimuth 

(Degrees) 

Dip  (Degress) 

1 

A 

1 

959.5 

0.9852 

O 

37mm 

0.19 

-6.0 

18.9 

2 

A 

2 

1.0 

B 

3 

A 

3 

1.0 

B 

4 

A 

4 

1.0 

B 

5 

A 

5 

1633.2 

0.9821 

O 

37mm 

0.11 

61.7 

9.8 

Figure  7-1 :  APG  Blind  Grid  report. 
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8.0  PERFORMANCE  ASSESMENT 


APG  Demonstration 

Overview 

The  SAINT  system  was  demonstrated  on  the  APG  Calibration  Grid  and  Blind  Grid  from 
June  22nd  to  July  8th,  2010.  A  total  of  12  days  was  spent  on  site.  The  first  day  was  spent 
flagging  the  Blind  grid  and  the  last  day  packing  up.  The  data  was  collected  by  NAEVA 
Geophysics  field  personnel.  The  field  operators  were  trained  on  the  SAINT  for  one  day 
prior  to  the  test;  they  were  already  familiar  with  the  EM61-HH  and  its  operation  with  the 
Allegro. 

Data  collection  was  carried  out  by  two  field  workers;  one  to  collect  data  and  the  other  to 
take  notes  and  assist  as  needed.  An  analyst  was  on  site  for  half  of  the  time  to  quality 
check  the  data  as  it  was  collected.  After  that,  the  data  was  transferred  nightly  to  an 
analyst  off  site  for  processing.  Typically,  data  would  be  downloaded  from  the  SAINT 
twice  daily,  at  lunch  and  the  end  of  the  day.  Batteries  would  be  swapped  then  as  well. 

The  only  major  problem  encountered  was  a  loose  cable.  Operations  had  to  be  shut  down 
for  !4  day  to  return  the  equipment  to  ENSCO  to  repair  the  cable  and  check  the  system 
out.  Besides  this,  every  10  or  20  data  files,  a  glitch  would  occur  in  the  raw  data  files.  A 
way  was  found  to  step  past  the  corrupted  data  and  process  the  rest  of  the  file.  Part  of  one 
afternoon  was  lost  to  a  thunderstorm. 

All  66  cells  in  the  Calibration  Grid  were  measured  by  the  SAINT.  On  the  Blind  Grid,  the 
EM61-HH  was  swept  over  the  cell  area.  If  no  measurable  signal  (~  10  mV)  was  noted  by 
the  operator,  the  cell  was  recorded  as  empty  and  no  SAINT  data  was  collected.  On 
average,  50  to  100  cells  were  measured  per  day. 

Template  Comparison 

As  a  baseline  comparison,  the  Calibration  and  Blind  Grids  were  also  measured  with  the 
EM61-HH  placed  on  a  wooden  template  grid.  Photos  of  both  the  template  and  SAINT 
measurements  being  made  at  APG  are  shown  in  Figure  8-1.  The  template  consisted  of  a  6 
by  6  square  grid  of  points  with  a  spacing  of  0.15  m.  Each  grid  had  38  measurements  with 
the  first  and  last  being  in  the  air  to  zero  the  sensor.  Template  measurements  took  roughly 
3-4  minutes  a  cell  with  a  comparable  average  daily  rate  to  the  SAINT.  Logistically,  the 
wooden  template  had  to  be  moved  cell  to  cell  which  was  comparable  to  moving  the 
SAINT  tripod  cell  to  cell.  Approximately,  10  grids  were  re-measured  due  to  operator 
error  in  taking  the  data  (grid  points  skipped  or  repeated).  Both  the  SAINT  and  template 
data  were  fit  with  the  same  EMI  model.  Discrimination  was  based  on  the  same  library  fit 
and  ratio  test.  Cells  that  were  noted  empty  for  the  SAINT  were  not  measured  by  the 
template.  Separate  target  reports  were  generated  and  handed  in  for  grading  with  both 
approaches.  As  with  the  SAINT,  a  set  of  data  quality  and  fit  quality  plots  were  generated 
for  each  cell.  Examples  of  these  are  shown  in  Appendix  B  and  C. 
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Figure  8-1  Photographs  of  the  EM61HH  plus  template  positioning  system  (top  photo)  and 
SAINT  positioning  system  (bottom  photo) 

Results 

The  plots  in  Figure  8-2  are  a  basic  comparison  of  the  EM61-HH  data  collected  with  the 
template  and  the  SAINT.  The  left  side  is  template  data,  and  the  right  side  is  SAINT  data. 
Both  were  collected  over  the  same  object  on  the  Blind  Grid.  The  top  plots  are  contours  of 
the  data  and  the  bottom  plots  are  the  data  as  sampled  over  time.  The  contoured  template 
data  has  symbols  plotted  at  each  measurement  point.  The  contoured  SAINT  data  has  a 
dotted  line  indicating  the  sensor  trajectory.  The  SAINT  data  has  been  time  shifted  to 
account  for  the  sensor’s  analog  filter.  The  template  data  is  not  affected  by  the  filter 
because  it  is  collected  with  the  sensor  stationary  at  each  point.  The  filter  also  stretches 
and  damps  the  signal  which  is  why  the  SAINT  signature  is  different  from  the  template 
signature.  The  signature  shown  is  typical  of  small,  shallow  clutter  measured  by  the 
EM61-HH.  Because  the  transmit/receive  coils  are  not  co-located,  there  is  a  region  where 
the  signal  actually  goes  negative.  With  six  continuous  slices  of  data  instead  of  six  sets  of 
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six  points,  the  SAINT  trajectory  collects  a  higher  density  of  data  points  and  better 
resolves  the  signature.  The  trade-off  is  larger  errors  in  positioning.  The  red  curves  in  the 
bottom  plots  are  the  modeled  fits  to  the  data. 


Samples  Samples 


Figure  8-2  Representative  data  acquired  using  a  template  (left  panels)  and  SAINT  (right 
panels).  Note  the  increased  data  density  acquired  by  the  SAINT  system. 

An  unexpected  outcome  of  EM61-HH  measurements  on  the  Calibration  Lanes  and  Blind 
Grid  at  APG  was  the  large  amount  of  very  small,  shallow  clutter  detected.  Both  sets  of 
measurements  were  taken  close  to  the  surface  (<  10  cm).  Combined  with  the  small  coils 
of  the  EM61-HH,  small  bits  of  shallow  metal  produce  very  large  signals.  Figure  8-3  is  an 
example  of  this  on  the  Calibration  Lanes.  The  top  plot  is  a  contour  of  the  data  in  black 
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with  the  SAINT  trajectory  in  red.  The  bottom  plot  is  the  signal  in  black  as  function  of 
samples  collected  over  time.  There  is  a  broad,  low  amplitude  signal  from  the  ordnance 
item  buried  here  and  a  large  spike  off  to  the  left  from  a  clutter  item.  In  this  case,  the 
clutter  data  could  be  weighted  out  of  the  inversion  process  and  a  reasonable  fit  to  the 
ordnance  signature  achieved.  This  model  fit  is  plotted  in  red  on  the  bottom  plot.  In  many 
cases,  this  was  not  the  case  and  the  item  would  be  categorized  as  “Can  Not  Analyze.” 
With  the  higher  density  of  data,  it  was  easier  to  resolve  these  clutter  situations  with  the 
SAINT  data  set,  but  many  clutter  spikes  were  noted  with  the  template  set  as  well.  In 
either  case,  the  EM61-HH  probably  detected  these  clutter  items  on  some  cells  that  the 
ground  truth  for  the  site  considers  to  be  empty  cells.  Since  this  is  not  a  detection  test,  but 
a  discrimination  study,  we  were  not  concerned  with  this  in  our  analysis  of  the  data. 
Anecdotally,  the  small  clutter  is  apparently  the  result  of  bringing  in  fill  dirt  that  was 
thought  to  be  clean,  but  turned  out  not  to  be. 


Figure  8-3  SAINT  positioned  data  showing  a  high  amplitude  anomaly  (-1.5m  X  top  and 
-110  samples  bottom)  that  is  superimposed  onto  the  broader,  lower  amplitude  signature 
of  the  test  target. 
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Separate  target  reports  were  handed  in  for  the  SAINT  and  template  data  sets.  For  the 
standardized  test  sites,  the  target  report  is  divided  into  a  detection  based  “response  stage” 
ranking  and  a  “discrimination  stage”  ranking.  For  the  response  stage,  the  EM61-HH  was 
manually  swept  over  the  center  of  each  cell.  If  there  was  no  measurable  signal  (~  10  mV) 
within  roughly  0.25  m  of  the  cell  center,  the  cell  was  determined  to  be  empty  for  both 
data  sets  and  no  further  measurements  were  taken.  The  search  was  limited  to  the  center  of 
each  cell,  because  (as  discussed  above)  there  was  a  lot  of  small  clutter  found  over  the  site. 
The  response  stage  factor  was  set  to  1.0  for  empty  cells  and  to  the  peak  measured  signal 
(first  time  gate)  on  cells  that  were  not.  For  “classification”,  the  empty  cells  are  labeled 
“B”  for  blank.  All  of  the  SAINT  and  template  data  with  measurable  signal  was  run 
through  the  dipole  model  based  inversion  and  a  standard  set  of  diagnostic  plots  generated 
for  each  cell  (see  appendices  B  and  C).  As  discussed  earlier,  the  data  was  inverted  with 
no  constraint  on  the  magnetic  polarizations  and  then  inverted  with  the  polarizations 
constrained  to  a  library  set  of  values  for  the  expected  ordnance  set.  The  ratio  of  the  fit 
error  (constrained  over  unconstrained)  was  used  as  the  discrimination  stage  ranking 
factor  and  the  library  entry  that  best  matched  was  used  to  declare  the  ordnance  type  in  the 
target  report.  This  discrimination  ranking  factor  varied  from  0.53  to  0.9998  for 
reasonable  data  sets.  Based  on  pre-test  data,  Calibration  Lane  data,  and  general 
observation  over  the  Blind  Grid  data,  a  threshold  of  0.98  was  empirically  selected  as  the 
threshold  between  declaring  the  “classification”  category  as  “C”  for  most  likely  clutter  or 
“O”  for  most  likely  ordnance.  Lastly,  the  diagnostic  plots  were  looked  through  by  the 
analyst  for  any  problems.  Cells  where  there  were  weak  signals,  multiple  objects,  or 
otherwise  poor  fits  were  labeled  as  “Can  Not  Analyze.”  The  standardized  target  reports 
do  not  account  for  this  category.  On  an  actual  site,  one  would  have  to  dig  these  items 
because  the  analysis  could  not  identify  them  with  certainty.  In  this  spirit,  the 
discrimination  stage  ranking  factor  for  these  items  was  simply  set  to  1.0.  They  are  all 
ranked  first  to  dig  in  no  particular  order.  There  were  44  of  these  items  in  the  SAINT 
target  report  and  41  in  the  template  target  report. 

The  two  target  reports  for  the  APG  Blind  Grid  were  submitted  for  grading  and  the 
returned  results  are  presented  in  Appendix  D.  An  overall  synopsis  is  shown  in  Table  8-1 
thru  Table  8-5  with  a  comparison  to  the  performance  of  the  advanced  5x5  TEM  array 
(MR-0601)  on  the  same  site. 

Table  8-1  presents  the  detection  results  of  the  EM61-HH  (same  for  SAINT  and  template) 
and  Table  8-2  presents  the  TEM  array  detection.  The  probability  of  detection,  Pd,  is 
broken  down  into  three  ordnance  categories  (105mm’s,  60mm/81mm,  and  25mm/37mm) 
as  well  as  three  depth  ranges  by  ordnance  diameter.  The  EM61-HH  missed  only 
105mm’s  (projectile  and  HEAT)  and  only  at  their  deepest  depth  range  (8-12  times  their 
diameter)  with  a  Pd  of  0.67  in  this  range.  All  other  P/s  were  1.0  with  an  overall  Pd  of 
0.99.  With  the  small  coils  of  the  EM61-HH  this  result  was  expected.  In  comparison,  the 
5x5  TEM  array  missed  fewer  of  the  deep  105mm’s  with  a  Pd  of  0.83  in  this  category. 
With  larger  coils  and  stationary  data  collection,  the  detection  of  the  5x5  TEM  array  is 
expected  to  be  better  for  deeper  ordnance  items. 
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Table  8-1.  EM61-HH  Response  Stage  P/s 


All  Types 

105-mm 

81/60-miri 

37/25-mm 

0.99 

0.98 

1.00 

1.00 

By  Depth 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

0.67 

0.67 

0.00 

1.00 

Table  8-2.  TEM  Array  Response  Stage  P/s 


All  Types 

105-mm 

81/60-mm 

37/25-mm 

0.99 

1.00 

1.00 

1.00 

By  Depth 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

0.78 

0.83 

0.00 

1.00 

Table  8-3  presents  the  discrimination  stage  P/ s  of  the  three  systems.  At  their  designated 
discrimination  stage  thresholds,  the  SAINT  results  are  equal  to  the  TEM  results  with  a  Pd 
of  0.99.  The  template  data  and  SAINT  data  were  processed  with  the  same  discrimination 
algorithm  (library  ratio  test)  with  the  same  threshold  setting,  but  the  template  data  only 
achieved  an  overall  Pd  of  0.93.  Table  8-4  compares  other  general  statistics  between  the 
systems.  The  second  column  is  efficiency  which  is  a  measure  of  how  much  of  the 
ordnance  detected  was  identified  as  ordnance  in  the  discrimination  stage.  In  this  column, 
the  first  number  is  the  efficiency  at  the  operator’s  threshold  and  the  second  is  the  highest 
efficiency  achieved  at  a  lower  threshold  level  (“without  loss  of  Pj')-  At  the  operator’s 
threshold  with  an  efficiency  of  1.0,  the  SAINT  system  identified  all  detected  ordnance  as 
being  ordnance.  The  TEM  efficiency  is  0.99,  but  achieves  1.0  at  only  a  slightly  lower 
threshold.  The  template  efficiency  is  0.92  at  the  threshold,  but  achieved  1.0  at  a  lower 
threshold  setting.  Between  discrimination  stage  Pd  and  efficiencies,  these  results  indicate 
that  all  three  systems  can  identify  ordnance  given  a  correct  setting  of  the  operator’s 
threshold. 


30 


Table  8-3.  Comparison  of  Discrimination  Stage  P/s 


Data  Set 

All  Types 

105-mm 

81/60-mm 

37/25-mm 

Template 

0.93 

0.94 

0.94 

0.98 

SAINT 

0.99 

0.98 

1.00 

1.00 

5x5  TEM  Array 

0.99 

0.98 

1.00 

1.00 

Table  8-4.  Comparison  of  Overall  Statistics 


Efficiency 

P cd 

P ba 

Response  Stage 

Pfr 

False  Positive 

Rejection  Rate 

P  ba 

Discrim  ination 
Stage 

Background  Alarm 

Rejection  Rate 

Template 

0.92/1.0 

1.0 

0.42 

0.35 

0.71/0.31 

0.14 

0.66/0.60 

SAINT 

1. 0/1.0 

1.0 

0.42 

0.43 

0.63/0.63 

0.16 

0.63/0.63 

5x5  TEM 
Array 

0.99/1.0 

0.90 

0.12 

0.03 

0.99/0.95 

0.05 

0.61/0.57 

In  terms  of  detecting  clutter,  the  EM61-HH  outperforms  the  TEM  array.  This  is  most 
likely  the  indication  of  small  metallic  clutter  present  on  the  Blind  Grid  as  already 
mentioned.  In  terms  of  emplaced  clutter,  the  EM61-HH  detected  all  of  it  with  a  Pcd  of 
1 .0.  The  TEM  array  Pcd  is  0.90.  On  “Blank”  cells  (no  emplaced  clutter  or  ordnance),  the 
EM61-HH  has  a  response  stage  probability  of  “background  alarm”,  P/M,  of  0.42  versus 
the  TEM  result  of  0.12.  The  EM61-HH  P ba  is  probably  a  good  indication  of  how  much 
small  metallic  debris  there  is. 

In  terms  of  rejecting  clutter  at  the  discrimination  stage,  the  TEM  array  outperforms  the 
SAINT  and  template  systems.  At  the  operator’s  threshold,  the  SAINT  data  resulted  in  a 
probability  of  false  positive,  Pfi»  of  0.43  and  the  template  data  gave  0.35.  The  TEM  array 
Pfp  is  0.03.  The  false  positive  rejection  rate,  a  measure  of  how  much  detected,  emplaced 
clutter  is  identified  as  clutter,  is  0.71  for  the  template  and  0.63  for  the  SAINT  data. 
Again,  at  the  template  threshold  for  a  Pd  of  1.0,  the  rate  is  0.31  compared  to  the  SAINT’s 
rate  of  0.63.  The  SAINT  data  is  significantly  better  at  rejecting  emplaced  clutter.  The 
TEM  array  outperforms  both  with  a  rejection  rate  of  0.95  at  a  Pd  of  1.0.  For  the 
background  alarm  rejection,  all  systems  performed  comparably  (~  0.60),  but  the  TEM 
detected  far  fewer  of  the  small  clutter  items  on  the  empty  cells. 

Table  8-5  presents  the  number  of  emplaced  ordnance  items  correctly  identified  by  type. 
This  number  is  at  the  operator’s  threshold.  Ordnance  not  detected  is  included  in  the  total 
count.  For  small  ordnance  (25mm/37mm),  the  SAINT  correctly  identified  100%  and  the 
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template  only  93%.  The  numbers  decrease  for  the  two  larger  ordnance  categories  (97% 
and  77%  for  the  SAINT).  The  TEM  array  numbers  are  listed  individually,  but  are  actually 
comparable  to  the  SAINT  performance.  It  should  be  noted  that  the  EM61-HH  library 
grouped  both  types  of  105mm’s  together.  The  TEM  array,  with  its  later  time  gates,  could 
distinguish  between  the  two  items. 

Table  8-5.  Comparison  of  Correct  Type  Classification 


25mm/3 

7mm 

60mm/81  mm 

105mm’s 

Overall 

Template 

93% 

87% 

73% 

84% 

SAINT 

100% 

97% 

77% 

91% 

5x5  TEM  Array 

100/100 

100/93 

70/67 

? 

To  better  understand  these  results,  we  compared  the  exact  target  reports  of  the  template, 
SAINT,  and  TEM  array  for  the  Blind  Grid.  To  analyze  the  template  and  SAINT  results, 
we  simply  used  the  TEM  array  data  as  ground  truth.  Overall,  the  TEM  array  missed  or 
was  wrong  on  only  a  small  number  of  items.  Rather  than  separate  clutter  into  emplaced 
and  background  alarms,  we  considered  both  as  just  one  set  of  clutter.  Lastly,  we 
considered  the  category  of  “Can’t  Analyze”  separately.  Figure  8-4  presents  a 
discrimination  stage  ROC  curve  based  on  this  work.  The  x-axis  is  just  a  straight  count  of 
clutter  items.  If  the  SAINT  data  set  or  template  data  set  detected  a  signal  on  a  non¬ 
ordnance  cell,  it  is  counted  as  a  valid  clutter  count,  even  if  the  TEM  detected  nothing  on 
that  cell.  The  y-axis  is  a  percentage  of  the  total  ordnance  based  on  what  the  TEM 
identified  as  ordnance.  The  black  curve  is  from  the  SAINT  set  and  the  red  curve  is  the 
template  set.  Items  in  the  “Can’t  Analyze”  category  have  been  ranked  first  in  the  ROC 
curve  (all  were  fixed  with  a  library  ratio  of  1.0),  but  plotted  on  the  negative  y-axis.  Both 
the  SAINT  and  template  data  had  comparable  numbers  in  this  category  (44  and  41)  and 
roughly  the  same  percentage  of  ordnance  occurred  here  (about  5%).  While  both  ROC 
curves  rise  rapidly,  the  SAINT  reaches  a  Pd  of  1.0  with  about  20  fewer  false  alarms. 
Ignoring  the  “Can’t  Analyze”  items,  the  SAINT  rejects  about  75%  (120/160)  of  the 
clutter  items  at  this  point.  The  template  data  rejects  about  63%  of  the  clutter  at  its  1.0  Pd 
point. 
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Number  of  FA 

Figure  8-4  Receiver  operating  characteristic  (ROC)  curve  comparing  classification 
performance  between  SAINT+EM61HH,  Template+EM61HH,  and  an  EM61  Mkll 
sensor. 

The  TEM  array  collected  data  on  the  Indirect  Fire  site  at  APG.  For  comparison,  a 
standard  EM61  Mkll  cart  system  was  deployed  and  analyzed  over  the  same  field  [REF 
7],  In  a  fashion  similar  to  the  SAINT/template  on  the  Blind  Grid,  we  considered  the  TEM 
array  results  as  ground  truth  and  created  the  blue  ROC  curve  in  Figure  8-4  for  the  EM61 
cart  results.  While  not  a  direct  comparison,  it  does  indicate  that  the  EM61-HH  positioned 
either  by  the  SAINT  or  the  template  performs  significantly  better  than  standard  EM61 
Mkll  data  on  a  survey  cart  with  GPS  positioning. 

Figure  8-5  and  Figure  8-6  plot  the  inverted  polarizations  for  the  items  that  the  TEM  array 
definitely  identified  as  ordnance.  The  polarizations  are  from  the  first  time  gate.  The  x- 
axis  indicates  the  primary  polarization.  The  y-axis  plots  the  second  and  third  values 
connected  by  a  vertical  line  and  a  symbol  at  their  average.  The  top  plots  in  Figure  5  are 
from  the  EM61  Cart  on  the  Indirect  Fire  field.  The  middle  plots  are  from  the  template 
data  on  the  Blind  Grid.  The  bottom  plots  are  from  the  SAINT  data  on  the  Blind  Grid.  The 
three  large  ordnance  types  are  shown:  60mm  on  left,  81mm  in  middle,  and  105mm  on 
right.  The  red  circles  are  centered  on  the  library  polarization  values.  The  EM61  Cart  plots 
clearly  show  that  this  data  cannot  be  used  to  accurately  invert  the  three  polarizations. 
Between  the  data  sparseness  (0.75  m  track  spacing)  and  the  positioning  errors  (GPS  RMS 
error  ~  2-3  cm),  one  can  at  best  use  the  primary  polarization  as  an  indication  of  size  (or 
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the  sum  of  the  polarizations  squared).  Even  looking  at  only  the  primary  polarization, 
there  is  considerable  overlap  between  the  three  ordnance  types.  Overall,  the  template  and 
the  SAINT  data  show  very  tight  clustering  of  the  polarizations.  The  three  ordnance  types 
can  be  clearly  distinguished  with  a  single  time  gate.  The  template  data  shows  some  extra 
spread  for  the  105mm  fits.  The  SAINT  has  several  individual  outliers  for  the  60mm  and 
105mm.  The  81mm  outliers  in  both  data  sets  are  clustered  and  based  on  TEM  analysis  are 
from  a  different  type  of  81mm  mortar  buried  on  the  Blind  Grid.  Figure  8-6  shows  the 
inverted  polarizations  from  SAINT  and  template  data  over  the  smaller  items:  25mm  and 
37mm.  There  are  no  EM61  Cart  results  for  these  small  items,  because  they  are  not 
present  on  the  Indirect  Fire  site.  For  the  small  ordnance,  both  data  sets  are  tightly 
clustered  and  there  is  no  uncertainty  as  to  the  ordnance  type.  Figure  8-7  plots  the  SAINT 
polarizations  of  items  that  the  TEM  data  indicates  are  clutter,  but  that  the  “library  ratio 
test”  processing  of  the  SAINT  data  flags  as  ordnance  (false  positives).  Many  of  these 
false  positives  are  from  small  clutter  items  with  polarizations  about  the  37mm  library 
value.  Given  the  spread  of  these  clutter  polarizations  relative  to  the  actual  37mm 
polarizations,  the  “library  ratio  test”  may  be  too  conservative  of  an  approach  for 
discriminating  the  inversion  results  of  SAINT  data.  It  may  actually  be  worth  comparing 
the  polarization  values. 
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SAINT  TEMPLATE  CART 


60mm  81mm  105mm 


0.1  1.0  10.0  100.0  1000.0  0.1  1.0  10.0  100.0  1000.0  0.1  1.0  10.0  100.0  1000.0 
Primary  Primary  Primary 


Figure  8-5  Principal  axis  polarizations  for  anomalies  that  were  declared  as  60mm,  81mm, 
and  105mm  by  NRL’s  TEM  sensor. 
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SAINT  TEMPLATE 


37mm  25mm 


0.1  1.0  „  .10.0  100.0  1000.0  0.1  1.0  „  .10.0  100.0  1000.0 
Primary  Primary 

Figure  8-6  Principal  axis  polarizations  for  anomalies  that  were  declared  as  37mm  and 
25mm  projectiles  by  NRL’s  TEM  sensor. 
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Secondaries  Secondaries 


0.1  1.0  10.0  100.0  1000.0  0.1  1.0  10.0  100.0  1000.0 
Primary  Primary 


Figure  8-7  Principal  axis  polarizations  for  anomalies  that  were  declared  clutter  by  NRL’s 
TEM  sensor. 
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9.0  COST  ASSESMENT 


Table  9-1.  Cost  Elements. 


Cost  Element 

Data  to  be  Tracked 

Estimated  Costs 

Instrument  Cost 

Component  costs  and  integration  costs 

•  Engineering  estimates  based  on 
current  development 

•  Spares  and  repairs 

EM61HH  -  $20k 

SAINT  -  $60k 
Template  -  $20 

Mobilization  and 
demobilization 

Cost  to  pack  the  equipment,  mobilize  to 
the  site,  and  return 

•  Derived  from  demonstration  costs 

$  lk 

Instrument  set-up  costs 

Cost  to  assemble  the  system  and  perform 
initial  calibration  tests. 

•  Personnel  required 

•  Hours  required 

$  lk 

Survey  costs 

Unit  cost  per  anomaly  investigated.  This 
was  calculated  as  daily  survey  costs 
divided  by  the  number  of  anomalies 
investigated  per  day. 

•  Daily  warm-up  and  calibration 

•  Survey  personnel  required 

•  Survey  hours  per  day 

•  Daily  equipment  break-down  and 
storage 

$14 

Discrimination  data 
processing 

Processing  costs  per  anomaly 

•  Personnel  required 

•  Time  required  per  anomaly 

$10 

9.1  COST  DRIVERS 

The  primary  costs  drivers  are  related  to  data  collection  (survey  costs)  and  to  data 
analyses.  Survey  costs  were  comparable  for  the  template-based  or  SAINT-aided 
EM61HH  systems.  As  demonstrated  here,  however,  repeat  data  collects  were  performed 
for  all  blind  targets  to  mitigate  an  unpredictable  spike  which  occasionally  occurred. 
Repeat  data  collects  will  not  be  needed  after  the  sensor  spike  issue  is  resolved.  Because 
both  data  were  analyzed  using  the  same  inversion  algorithms,  the  analysis  costs  were  also 
comparable. 
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9.2  COST  BENEFIT 


The  realized  cost  benefit  for  any  given  site  depends  greatly  on  the  actual  costs  required 
for  excavation.  The  SAINT+EM61HH  system  had  discrimination  Pd  of  0.99  while 
rejecting  0.63  of  the  clutter.  The  template+EM61HH  system  was  only  slightly  poorer. 
These  results,  if  adopted  and  proven  reliable  going  forward,  would  allow  the 
stakeholder(s)  to  uniquely  address  suspect  UXO  while  leaving  over  60  percent  of  the 
objects  in  the  ground. 
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10.0  IMPLEMENTATION  ISSUES 


We  collected  data  with  an  EM61HH  sensor  using  two  different  methods  for  spatially 
registration.  The  two  methods  included  a  simple  template  and  the  SAINT  system.  No 
significant  problems  or  issues  were  noted  by  the  field  crew  while  acquiring  either  data 
set.  Because  of  an  irregular  and  unpredictable  data  spike  associated  with  the  SAINT 
positioning  system,  we  acquired  data  at  each  target  location  multiple  times.  Although 
this  approach  maximized  data  quality  because  we  could  discard  problematic  data,  it  did 
slow  down  production.  Aside  from  this,  there  were  no  implementation  issues  of  note. 
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APPENDIX  A:  SYSTEM  IMPROVEMENTS  AND  VALIDATION 

TESTS 


The  improved  system  is  shown  in  Figure  A-l.  The  system  is  enclosed  in  the  triangular, 
ABS  plastic  case  with  handle  that  can  be  attached  to  an  EMI  sensor  pole.  The  SAINT 
was  again  attached  to  an  EM61-HH  Mkll,  but  options  were  kept  open  for  attaching  it  to 
handheld  systems  under  development.  The  case  contains  a  single  board  computer  (SBC), 
an  electronic  compass,  and  an  IMU  (Honeywell  1900).  The  EM61-HH  is  controlled  by 
the  standard  Allegro  handheld  computer  running  software  provided  by  Geonics.  The 
combined  instruments  sit  on  an  improved  tripod  stand. 


Allesro  CX 
SAINT 

EM-61  HH 
Tripod 


Figure  A-l  The  SAINT  system  installed  on  an  EM-61  HH  sensor 

The  IMU  data  collection  was  changed  from  a  simple  embedded  processor  to  a  single 
board  computer.  This  was  done  to  make  the  system  more  versatile.  The  SBC  has  a  variety 
of  digital  input/output  options  (serial  lines,  USB  ports,  etc.).  A  serial  line  is  used  to 
provide  synchronization  of  the  IMU  data  with  the  EM61  data.  The  serial  line  grabs  the 
trigger  characters  sent  from  the  Allegro  to  the  EM61  as  well  as  the  data  packets  sent  back 
from  the  EM61  to  the  Allegro.  Each  set  is  time  stamped  with  the  count  of  the  most  recent 
IMU  packet  received.  The  IMU  packets  come  in  at  a  rate  of  600  per  second.  The  EM61 
data  is  time  stamped  to  within  a  small  number  of  IMU  counts  (~3-5)  and  a  timing 
accuracy  of  <  10  milliseconds.  For  other  EMI  systems,  the  various  EO  lines  could  be 
used  to  time  stamp  other  EMI  data  streams  or  to  provide  a  timing  pulse  out  to  other  EMI 
systems.  The  prototype  system  had  no  synchronization  of  the  EMI  and  IMU  data. 

The  IMU  and  EM61  data  are  stored  in  files  on  an  SD  memory  card  on  the  SBC.  The  data 
is  moved  to  a  PC  computer  for  post-processing.  To  transfer  the  data  files,  the  SBC  has 
been  set  up  to  emulate  a  USB  memory  device.  By  switching  the  SAINT  system  to  a  USB 
cable,  it  plugs  into  a  PC  and  automatically  mounts  as  an  external  drive.  The  files  can  be 
dragged  and  dropped  from  the  SAINT  to  the  PC.  The  prototype  system  had  a  compact 
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flash  card  that  had  to  be  physically  removed  and  plugged  into  a  PC  with  a  flash  card 
reader. 

Figure  A-2  shows  a  close  up  of  the  SAINT  case.  There  is  a  green  LED  push  button  for 
powering  the  system  up,  pausing  data  collection,  and  powering  the  system  off.  There  are 
two  status  LED’s  on  top  of  the  case.  The  blue  one  provides  status  and  timing  flashes 
during  data  collection  and  file  download.  The  red  LED  indicates  system  errors.  Errors 
and  timing  are  also  indicated  by  an  audible  tone.  Cabling  for  power,  downloading  data, 
serial  lines,  and  the  tripod  switch  attach  to  the  side  of  the  case.  Further  detail  of  cabling, 
operations,  and  LED  status  lights  is  given  in  the  SAINT  user  manual  [REF  5].  Figure  A-3 
shows  the  battery  pack  for  the  SAINT  attached  to  the  EM61  backpack.  The  new  system  is 
powered  by  a  laptop  style,  lithium  ion  battery.  It  is  capable  of  running  the  SAINT  for 
over  8  hours.  There  is  a  spare  battery  and  an  easy  to  use  charger.  In  the  field,  the  battery 
was  usually  changed  at  the  mid-day  break  to  insure  no  possible  loss  of  power.  The  battery 
pack  has  its  own  on/off  button  with  a  red  LED  to  indicate  that  the  battery  pack  is 
providing  power  to  the  SAINT. 


12-pin  Receptacle 

(Cable-A  or  Cable-B) 


Red  &  Blue  LEDs 


Push-button  Switch 
with  Green  LED 


10-pin  Receptacle  (Cable- 
C  or  Cable-D) 


Figure  A-2  The  SAINT  system  IMU  housing  with  interface  details 
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EM61  Back  Pack 


SAINT  Battery  Pack 


Figure  A-3  The  SAINT  system  battery  pack  strapped  to  the  EM-61  backpack 

The  new  system  replaced  the  all  metal  tripod  used  with  the  prototype.  The  legs  on  the 
new  tripod  are  all  non-metallic.  The  top  cone  on  the  tripod  where  the  SAINT  sits  was 
broadened  to  ease  the  placement  of  the  system  on  the  tripod  (see  Figure  A-4).  The  cup  on 
the  SAINT  that  sits  on  the  tripod  cone  has  a  ruggedized  switch  to  trigger  the  SAINT 
timing  when  it  is  lifted  on/off  of  the  tripod.  There  is  also  a  notch/key  on  the  cup/cone  to 
re-align  the  SAINT  when  it  is  placed  back  on  the  tripod.  One  leg  of  the  tripod  has  a 
footer  for  the  EMI  coil  to  rest  on  (see  Figure  A-5).  Figure  A-6  shows  the  EM61  coil  with 
a  bracket  made  to  fix  the  coil  orientation  relative  to  the  rest  of  the  SAINT/EM61  system. 


Figure  A-4  The  SAINT  system  tripod  -  base  plate  details 
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Fixed  length  leg 
Coil  Platform 

Adjustable  legs 


Figure  A-5  The  SAINT  system  tripod  -  leg  details 


Coil  cable 

Clamp-end 
of  bracket 


Coil  locking 
bracket 


Tie-straps 


Figure  A-6  The  SAINT  system  coil-locking  bracket 


Figure  A-7  illustrates  the  general  operation  of  the  SAINT.  The  photos  (from  left  to  right) 
show  it  being  used  on  a  wooden  platform  designed  for  general  testing.  In  general,  the 
tripod  and  SAINT  are  placed  within  a  meter  or  two  of  an  anomaly  location.  The  SAINT 
and  EM61  are  powered  up  on  the  tripod.  After  sitting  idle  for  15  seconds  of  background 
data,  the  SAINT  beeps  once  and  flashes  the  blue  LED.  The  user  lifts  the  system  and 
proceeds  to  sweep  the  sensor  back  and  forth  over  the  object  location.  The  SAINT  beeps 
and  flashes  at  several  intervals  during  data  collection.  After  30  seconds  of  collection  it 
beeps  and  lights  up  continuously  until  replaced  on  the  tripod.  After  another  15  seconds  of 
being  idle,  it  beeps/flashes  once  to  indicate  that  it  is  ready  to  move  again.  Examples  of 
actual  sweep  data  are  shown  below  the  photos  in  Figure  7. 
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Figure  A- 7  Standard  survey  procedures  include  taking  the  sensor  off  of  the  tripod, 
waving  it  above  the  targets’  location,  and  returning  the  sensor  to  the  tripod  for  a  zero 
velocity  update  (top  photographs).  The  two  scatter  plots  on  the  bottom  show  the  spatial 
trajectory  of  the  sensor  head  (bottom  left)  and  the  measured  EMI  data  (bottom  right). 

After  the  idle  period,  more  data  can  be  collected  over  the  same  object  or  from  nearby 
objects  without  moving  the  tripod.  All  of  the  IMU  and  EMI  data  would  go  into  the  same 
file  on  the  SAINT.  The  SBC  creates  a  new  file  on  startup  using  the  nomenclature 
“FILE_XXX.DAT”,  where  XXX  is  the  current  file  number.  The  file  number  is  derived 
from  a  configuration  file  on  the  SAINT  called  “SNTCNF.TXT”.  This  text  file  has  the 
current  file  number  on  it.  After  a  file  is  closed,  the  current  number  is  incremented.  After 
collecting  either  a  single  or  multiple  runs  in  a  file  the  system  can  be  either  paused  or 
shutdown  to  close  out  the  current  data  file.  To  pause  the  SAINT,  the  power  button  is 
pressed  once  briefly.  The  blue  status  LED  and  the  green  LED  on  the  power  button  will 
flash  on/off  continuously  to  show  that  it  is  paused.  The  SAINT  and  tripod  can  then  be 
moved  to  a  new  location.  To  start  at  the  new  location  with  a  new  file,  the  power  button  is 
pressed  again  briefly.  To  shut  the  system  down,  the  power  button  is  pressed  and  held 
until  the  SAINT  emits  a  continuous  tone.  At  this  time  the  battery  pack’s  power  button 
should  be  pressed  to  shutdown  the  power  to  the  SAINT. 


The  SAINT  data  files  are  stored  by  the  SBC  on  a  mini-SD  card.  When  started  with  its 
USB  cable,  the  SAINT  emulates  an  external  USB  drive  on  a  PC.  The  data  files 
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(“FILE_XXX.DAT”)  can  be  transferred  to  the  PC.  To  process  the  IMU  data  into  sensor 
trajectorys,  ENSCO  has  written  an  interactive  GUI  program  in  Matlab.  From  the  GUI, 
one  selects  the  data  file  to  process.  The  data  is  read  in  and  displayed  with  the  idle  periods 
identified  as  ZUPT’s  (Zero- Velocity  Updates).  An  example  is  shown  in  Figure  A-8.  The 
user  selects  which  periods  of  motion  between  ZUPT’s  to  process.  The  trajectories  are 
calculated  and  dumped  into  ASCII  files  called  “FILE_XXX.DAT_imuX.txt”.  The  second 
X  is  a  number  indicating  which  run  within  the  file  has  been  processed  (in  the  case  of 
multiple  runs  in  a  file).  The  “trigger”  times  of  the  EM61  by  the  Allegro  computer  and  the 
return  times  of  EM61  data  packets  are  placed  in  a  file  called  “FILE_XXX.trig.dat”.  The 
EM61  data  packets  are  placed  in  a  file  called  “FILE_XXX.em61.dat”.  The  trigger 
information  and  EM61  data  files  are  for  the  entire  file.  These  text  data  files  are  analyzed 
by  SAIC  using  routines  written  in  an  analysis  package  called  IDL.  The  routines  read  in 
the  data  and  automatically  match  up  the  EM61  times  to  the  IMU  position  times. 
Interactive  routines  trim  out  the  EM  data  during  a  specific  run.  This  data  is  passed  to  EMI 
inversion  routines  for  EM61-HH  data.  The  data  and  fit  results  are  plotted  out  in  a 
standard  set  of  diagnostic  plots. 


Figure  A-8  Select  the  ZUPT  periods  (identifiable  by  a  flat  line).  In  this  example,  there 
are  three  stationary  periods  marked  by  arrows  that  should  be  selected. 

Initial  Testing 

Shake  Down  of  System 

The  system  was  built  by  ENSCO  and  delivered  to  SAIC  in  January  of  2009.  From 
January  to  May  of  2009,  the  system  was  put  through  a  number  of  controlled  tests.  A 
variety  of  minor  glitches  were  found  and  corrected  in  the  hardware/software  of  the  SBC 
and  in  the  post-processing  software.  Initially,  there  were  a  number  of  problems  with 
corrupted  data.  Collecting  the  IMU  packets  at  600  Hz,  the  electronic  compass  data,  the 
EMI  triggers  and  the  EMI  data  packets  was  not  as  easy  as  ENSCO  thought.  The  status 
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lights  were  also  not  operating  properly.  It  took  several  iterations  between  controlled  tests 
and  returning  the  equipment  to  ENSCO  to  correct  the  problems  found. 

Figure  A-9  illustrates  one  of  the  simple  controlled  tests.  A  set  of  four  poles  were  set  out 
in  a  rectangle  at  a  known  distance  from  the  tripod.  The  sensor  was  lifted  from  the  tripod, 
moved  to  the  rectangle  and  would  touch  each  of  the  four  poles.  This  was  done  repeatedly 
in  a  single  data  collection.  The  plot  on  the  left  shows  the  x,y  trajectory  of  the  sensor  for 
repeated  runs.  Alternating  runs  are  plotted  in  red  and  black.  The  center  of  the  tripod  is  at 
(0,0)  and  the  distance  to  the  rectangle’s  lower  left  comer  is  plotted  as  a  dotted  arc. 
Initially,  there  were  problems  with  the  electronic  compass  data  and  in  post-processing 
calculations  of  IMU  bias  levels  during  the  motionless  ZUPT  periods.  From  ran  to  run  and 
to  a  lesser  degree  during  the  run,  the  heading  would  drift.  The  plot  on  the  right  shows 
data  collected  after  these  problems  were  fixed. 


Figure  A-9  Controlled  test  to  validate  system  registration.  The  test  protocal  included 
repeat  movements  to  four  poles  arranged  in  a  rectangle.  The  image  on  the  left  indicates  a 
problem  with  the  internal  compass.  The  data  on  the  right  were  acquired  after  the  problem 
with  the  compass  was  resolved. 

In  order  to  track  the  position  and  orientation  of  the  EMI  sensor  coil,  the  SAINT  system 
must  know  the  relative  location  and  orientation  of  the  coil  center  from  the  IMU.  Roll  and 
pitch  angles  were  checked  by  collecting  controlled  data  on  a  flat  surface.  The  system 
would  be  lifted  from  a  tripod,  placed  on  the  flat  surface  and  then  pitched/rolled  in 
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controlled  directions.  The  zero  level  and  sign  of  the  pitch/roll  angles  could  be  checked  in 
this  fashion.  To  check  the  yaw  angle  and  (x,y,z)  displacement  settings,  a  chalk  circle  was 
drawn  on  the  flat  surface.  Distances  and  angles  were  carefully  measured  from  the  tripod 
and  the  starting  location  of  the  sensor  coil  to  the  circle.  A  set  of  fixed  angles  were  marked 
(30,  60,  90,  120,  150  degrees).  The  coil  was  lifted  from  the  tripod,  moved  straight 
forward  to  the  circle  and  then  swept  along  the  circle  stopping  at  the  marked  angles. 
Figure  A- 10  plots  these  trajectories  with  the  original  (left  plot)  and  corrected  (x,y,z) 
displacements  (right  plot).  The  black  solid  curve  is  the  coil  center  trajectory.  The  red 
dashed  curve  is  the  circle  that  the  coil  center  was  meant  to  track.  The  inner  dotted  circle 
is  the  actual  chalk  circle  that  the  operator  traced  with  the  back  of  the  EMI  coil,  because 
the  operator  can  not  actually  see  the  coil  center.  The  dotted  rays  are  the  fixed  angles  that 
the  sensor  paused  at. 
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Figure  A- 10  Controlled  test  conducted  on  a  flat  surface  to  check  the  pitch,  roll,  and  yaw 
angels  as  well  as  the  displacement  settings.  See  text  above  for  details. 

The  SBC  times  everything  by  the  current  IMU  data  packet.  The  EM61-HH  is  triggered 
by  the  Geonics  software  running  on  a  handheld  Allegro  computer.  The  SBC  monitors  and 
time  stamps  the  serial  trigger  sent  by  the  Allegro  as  well  as  the  data  packet  sent  back  to 
the  Allegro  from  the  EM61-HH  electronics.  To  check  for  accurate  time  stamping  by  the 
SBC,  the  EM61-HH  data  on  the  Allegro  computer  was  compared  to  the  same  packets 
captured  on  the  SBC.  After  initial  fixes  to  correct  missed  and  corrupted  EM61-HH  data, 
the  timing  of  the  two  data  sets  matched  reasonably  well.  The  top  plot  in  Figure  A- 11 
matches  the  data  as  a  function  of  sample  number  between  the  packets  recorded  by  the 
SBC  (in  black)  and  the  packets  recorded  by  the  Allegro  (in  red).  The  data  shows  the 
system  being  swept  over  an  object  three  separate  times.  The  two  plots  below  this  graph 
the  time  steps  between  data  packets  as  recorded  by  the  two  systems.  The  data  rate  was 
roughly  15  samples  per  second  (0.067  s).  The  Allegro  software  (third  plot,  in  red)  plots 
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time  steps  to  an  accuracy  of  0.01  s.  There  are  occasional  long  time  steps  on  the  order  of 
0.14  s.  The  SBC  (second  plot,  in  black)  should  be  able  to  time  stamp  the  data  to  within 
several  IMU  time  counts  at  l/600th  of  a  second  (0.0017  s).  The  observed  jitter  in  time 
steps  is  much  larger.  It  does  match  the  Allegro  on  most  of  the  larger  time  steps.  It  would 
appear  that  every  now  and  then  the  Allegro  misses  sending  a  trigger.  However,  the  SBC 
also  has  several  other  large  steps  not  recorded  by  the  Allegro  where  triggers  are  being 
either  missed  or  incorrectly  time  stamped.  After  further  work  on  the  SBC  software,  better 
results  were  achieved.  The  last  two  plots  show  fixed  SBC  (black)  and  Allegro  (red)  time 
steps.  Better  results  were  achieved  on  the  Allegro  timing  by  using  newer  software  for 
Geonics  EM61  systems  produced  by  a  company  named  Geomar.  The  Geomar  software 
could  time  tag  the  data  to  within  0.001  s  with  jitter  on  the  order  of  several  samples.  There 
were  fewer  large  time  steps  in  the  data  as  well.  The  improved  SBC  software  showed  jitter 
on  the  order  of  several  IMU  data  packets,  0.0017  s.  It  reasonably  matches  any  larger  time 
steps  on  the  Allegro  and  had  few  if  any  glitches  not  matched  by  the  Allegro.  Overall,  the 
IMU  and  EMI  data  streams  are  matched  to  within  5-10  milliseconds. 
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Figure  A- 11  Raw  time  series  as  timed  by  a  SBC  and  logged  by  the  SAINT.  See  text 
above  for  additional  information. 
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Sweep  Pattern  Testing 

Concurrent  with  the  shake  down,  testing  was  done  to  determine  if  different  sensor  sweep 
patterns  would  collect  better  data  for  EMI  inversion.  For  the  operator,  the  easiest  sweep 
pattern  is  to  just  move  the  sensor  back  and  forth,  side-to-side.  Within  the  constraint  of  30 
seconds  for  data  collection,  it  is  possible  to  obtain  5-6  slices  of  data  over  a  buried  object. 
Because  the  EM61-HH  is  a  bistatic  sensor  (transmit  and  receive  coil  not  co-located),  a 
buried  object  would  be  better  covered  by  the  sensor  if  it  passes  over  the  object  in  several 
directions. 

Figure  A- 12  illustrates  a  simple  back  and  forth  sweep  in  the  upper  left  plot.  A  multiple 
direction,  “X”  pattern  sweep  is  shown  in  the  upper  right.  Other  sweep  patterns  were 
tested  as  well,  but  most  were  not  easily  performed  by  the  operator  in  the  time  allowed. 
Note  that  in  the  simple  sweep  pattern,  the  heading  of  the  sensor  is  kept  relatively  constant 
(along  the  y-direction).  With  the  “X”  pattern,  there  are  3  similarly  oriented  passes,  but  the 
two  “X”  passes  are  oriented  at  ±45  degrees  to  the  y-direction.  The  data  collected  by  these 
sweeps  are  shown  as  a  function  of  time  in  the  plots  below.  The  symbols  in  the  upper  plots 
are  the  x,y  locations  of  the  measured  EMI  data  plotted  with  the  same  symbols  in  the 
lower  plots.  The  red  curves  in  the  lower  plots  are  the  best  model  fits  to  the  data  from  the 
EMI  inversion. 


-1.0  -0.5  0.0  0.5  1.0  1.5  -1.0  -0.5  0.0  0.5  1.0  1.5 

X  (m)  X  (m) 


Figure  A- 12  Example  data  set  showing  the  spatial  sweep  pattern  (top)  and  rastorized 
EM61  HH  data  (bottom).  The  two  data  streams  are  merged  to  create  a  spatially  registered 
data  file. 
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SAINT  data  was  collected  multiple  times  over  the  same  object  with  these  two  sweep 
patterns.  The  data  was  inverted  using  standard  EMI  dipole  modeled  based  algorithms. 
The  model  parameters  of  location,  depth,  and  magnetic  polarizations  (“betas”)  for  each 
time  gate  were  obtained.  The  expectation  was  that  a  more  consistent  set  of  inverted 
parameters  would  be  obtained  with  the  “X”  pattern  sweep;  since,  this  pattern  collected  a 
more  diverse  set  of  EMI  data  from  the  object.  The  exact  opposite  of  this  was  observed. 
Figure  A- 13  plots  an  example  of  this  result.  These  plotted  fit  results  are  from  eight  simple 
sweeps  and  eight  “X”’ing  sweeps  measured  over  a  four  inch  steel  sphere  roughly  0.28  m 
below  the  test  stand.  The  black  plotted  symbols  are  the  inversion  results  from  a  simple 
sweep  and  the  red  symbols  are  for  the  “X”  one.  The  left  plot  shows  the  fitted  primary 
beta  response  versus  the  fitted  depth.  The  right  figure  shows  the  secondary  versus 
primary  beta  responses;  for  a  sphere,  these  should  be  equal.  The  observed  spread  is 
significantly  larger  for  the  “X”  pattern  sweeps. 


Fit  Depth  (m)  Primary  b  Response 


Figure  A- 13  Comparison  of  polarizations  derived  for  8  repeated  measurements  of  a 
common  object  for  simple  sweeps  and  X  sweep  patterns. 

Sweep  Pattern  Simulation 

Without  a  way  to  determine  the  true  trajectory  of  the  SAINT  system,  it  is  difficult  to 
diagnose  the  unexpected  results.  Past  work  has  shown  that  EMI  inversion  is  very 
sensitive  to  position  errors  [REF  6],  but  little  has  been  done  to  consider  different  types  of 
position  error.  One  suspicion  was  that  the  inherent  nature  of  IMU  based  positioning  was 
having  an  effect.  Because  the  IMU  measurements  are  integrated  to  determine  the  sensor’s 
trajectory,  the  position  errors  tend  to  be  a  systematic  drift  from  the  true  sensor  position. 
GPS,  RF  ranging,  and  laser  ranging  are  more  likely  to  produce  random  errors  about  the 
true  sensor  trajectory.  To  test  this  idea,  measured  sensor  trajectories  were  used  in  a 
simulation  to  compare  the  effect  of  the  two  different  position  errors  on  EMI  inversion 
results. 
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A  set  of  measured  simple  sweeps  and  “X”  sweeps  were  used  as  true  trajectories  in  a 
simulation.  For  each  run  of  the  simulation,  one  of  these  sweeps  was  randomly  chosen.  A 
random  x,y  object  location  was  generated  near  the  center  of  the  sweeps  (within  0.05  m). 
The  forward  model  for  the  EMI  sensor  was  run  for  this  trajectory  and  location  with  a 
fixed  depth  of  0.3  m  and  a  set  of  equal  (sphere)  beta  polarizations  of  14.0  m3.  Position 
noise  would  be  added  to  the  “true”  trajectory  and  then  the  data  would  be  inverted. 

Three  types  of  noise  were  added  to  the  trajectories:  no  noise,  random  noise,  and  a 
systematic  drift  that  varied  over  time.  The  random  noise  was  generated  with  a  random 
number  generator  with  a  standard  deviation  of  0.005  m.  These  random  shifts  were  added 
in  the  x,  y,  and  z  directions.  Different  magnitudes  of  random  error  were  tested,  but  the 
final  level  used  was  based  on  past  work  that  indicates  SAINT  position  errors  of  less  than 
one  centimeter  overall  [REF  4],  The  top  plot  in  Figure  A- 14  shows  an  example  sweep 
without  (as  a  black  curve)  and  with  random  position  errors  (red  symbols).  The  drift  errors 
were  generated  by  taking  5  points  along  the  trajectory  evenly  spaced  over  time.  A  set  of 
random  displacements  were  generated  for  each  of  these  five  locations.  The  magnitudes  of 
these  displacements  were  fixed  to  be  less  than  0.01  m.  From  these  five  shifts,  a  spline  fit 
was  done  for  dx,  dy,  and  dz  as  a  function  of  time.  These  splined  shifts  would  be  added  to 
the  true  x,  y,  z  trajectory.  The  lower  plot  in  Figure  1  shows  an  example  of  this  drifted 
trajectory. 


Figure  A-14  Graphical  display  of  a  sensor  trajectory  without  positional  noise  (black), 
with  random  deviations  (red  top),  and  with  drifting  (red  bottom). 


For  each  type  of  noise,  the  simulation  was  run  several  hundred  times.  For  each  run  the 
inversion  was  carried  out  over  a  range  of  fixed  depths  and  the  chi-square  value  and  RMS 
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fit  error  recorded.  These  “Goodness-of-Fit”  curves  as  a  function  of  depth  are  a  good 
indication  of  how  well  the  fit  algorithm  with  the  data  it  is  given  will  converge  to  the 
correct  result.  Steeper  curves  converge  well;  curves  with  multiple  minima  can  converge 
to  the  wrong  answer.  Examples  of  these  curves  are  shown  in  Figure  A- 15.  Each  plot 
shows  the  curves  for  ten  different  runs  of  the  simulation.  The  left  plots  are  based  on 
simple  sweeps  and  the  right  plots  are  from  “X”  sweeps.  The  top  set  are  from  simulation 
runs  with  no  position  errors.  With  no  errors,  the  both  types  of  sweeps  has  the  correct 
global  minimum  at  a  depth  of  -0.3  m,  but  the  “X”  sweeps  are  steeper,  and  the  simple 
sweeps  show  signs  of  secondary  minimums.  The  middle  sets  of  plots  are  with  random 
position  errors.  The  curves  are  broader  and  the  global  minimum  is  not  always  exactly  at  - 
0.3  m.  Again,  the  simple  sweeps  are  not  as  good  and  show  distinct  secondary  minima. 
The  bottom  set  of  plots  are  with  drift  position  errors.  In  this  case,  the  simple  sweeps  are 
the  better  set  of  curves;  the  global  minima  are  close  to  -0.3  and  the  secondary  minima  are 
no  worse  than  before.  The  “X”  sweeps  show  global  minima  over  a  larger  range  as  well  as 
some  secondary  minima.  The  fit  error  at  some  of  the  minima  has  also  become 
significantly  larger. 


Figure  A- 15  Goodness  of  fit  curves  for  two  surveys  and  three  distinct  noise  scenarios. 
There  are  ten  runs  (curves)  for  each  case  -  see  text  above  for  additional  details. 
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Figure  A- 16  plots  the  best  fit  parameters  for  these  different  position  error  cases.  The 
symbols  plot  the  fitted  depth  and  fitted  primary  beta  polarization  for  one  hundred  runs  of 
the  simulation.  Again,  the  left  plots  are  for  simple  sweeps  and  the  right  plots  are  “X” 
sweeps.  The  top  plots  are  with  random  position  errors  and  the  bottom  for  drift  errors. 
With  random  errors,  the  “X”  pattern  produces  a  smaller  spread  in  the  fit  parameters,  but 
for  drift  errors  a  sweep  pattern  gives  the  better  result.  While  these  spread  patterns  are  not 
exactly  equivalent  to  the  measured  results  in  Figure  2  (of  previous  section),  they  do  seem 
to  indicate  that  drift  errors  are  the  source  of  the  SAINT  system  EMI  inversion  spread  and 
that  a  simple  sweep  pattern  is  more  robust  in  the  face  of  these  errors.  In  general,  any  EMI 
data  collection  and  inversion  will  be  more  sensitive  to  systematic  position  errors  than 
random  ones. 
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Figure  A-16  Fitted  parameters  for  two  sweep  options  and  two  types  of  position  errors. 
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Test  Stand  Measurements  of  APG  Targets 


To  prepare  for  a  demonstration  of  the  SAINT  at  the  Aberdeen  Proving  Ground  (APG) 
Standardized  UXO  Technology  Test  Site,  a  set  of  the  possible  UXO  target  items  was 
obtained.  Because  the  SAINT  system  is  designed  for  local  interrogation  of  items  already 
detected,  it  was  decided  to  take  the  SAINT  to  the  Blind  Grid  scenario  only.  There  are  six 
possible  UXO  items  buried  there:  25mm,  37mm,  60mm  mortar,  81mm  mortar,  105mm, 
and  105mm  HEAT.  These  are  shown  in  Figure  A- 17.  These  items  were  placed  under  a 
wooden  test  stand  and  measured  repeatedly  at  different  depths  and  orientations. 


Figure  A- 17  Photograph  of  the  six  munitions  at  APG 

A  simple  sweep  pattern  was  used  and  repeated  four  times  to  check  for  variability  in  the 
EMI  inversion  results.  It  was  found  that  the  variability  could  be  minimized  by  keeping 
the  collection  time  to  a  minimum  and  by  limiting  the  amount  of  change  in  the  sensor 
heading.  This  resulted  in  a  very  fixed  data  collection  routine  as  illustrated  in  Figure  A- 18. 
The  tripod  would  be  placed  about  1  m  to  the  right  and  1.5  m  away  from  the  “flagged” 
location.  The  SAINT  system  would  be  lifted  from  the  tripod  and  moved  straight  forward. 
It  would  then  be  moved  left/right  back  and  forth  in  a  series  of  six  passes  without 
significantly  changing  the  sensor  heading.  On  arbitrary  surfaces  such  as  the  test  stand  and 
grassy  fields,  it  was  found  to  be  useful  to  place  a  marked  board  on  the  ground.  The  board 
had  marks  every  0.12m.  This  helped  guide  the  operator  and  insured  uniform  coverage 
over  the  flagged  spot.  Each  collection  was  on  the  order  of  15-20  seconds.  With  15 
sections  of  stationary  data  before  and  after  each  collection,  total  time  for  four  data 
collections  over  an  object  was  about  2-3  minutes. 
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Figure  A- 18  Schematic  illustrating  the  data  collection  concept  of  operations. 

Figure  A- 19  shows  the  result  of  multiple  sets  of  measurements  over  a  60mm  mortar  at  a 
variety  of  depths  and  orientations.  The  top  plot  shows  the  3  beta  polarizations  (1st  time 
gate)  inverted  from  the  data  as  a  function  of  data  run.  The  black  symbols  are  the  primary 
beta  and  the  red/green  symbols  are  the  secondaries.  The  dashed  lines  indicate  the  average 
value.  The  bottom  plot  shows  the  fitted  betas  on  a  log-log  scale.  The  symbols  are  plotted 
at  the  primary  beta  value  and  the  average  of  the  secondary.  The  line  through  each  symbol 
represents  the  range  of  the  two  secondaries.  The  average  values  are  shown  as  dotted  lines. 
Overall,  there  is  reasonable  clustering,  but  outliers  remain. 


58 


20 


|  15 

4— > 

#N 

'Vh 

Cj 

o  10 

PLh 

_Q 

<D 


.  o  O  °  oo  O 

to° . . o . oO . o . 

oO  O  O  o  o 


o°  o 
o  o 

o . Oo . 


Primary  b  -  black 
Secondary  b  rs  -  red/green 

o°  o 

OO  o 


O  oo  o  o 
O  O  .  v  x 

Oo  o  X 

(  Ox 

X^XXXv  V  „  °XO  -XX 


^X.^x4x.^y: . &  £  ~  x  x^  %  x^x  x^^xx  x^x^xx^X)<xxxx 

^  X  XX  X  X  v> 

x  X 


X 

*X  £ 


10 


20  30 

Data  Run  # 


40 


50 


0.01  0.10  1.00  10.00  100.00 
Primary  b 

Figure  A- 19  Fitted  parameters  for  multiple  60mm  mortars  at  a  variety  of  depths  and 
orientations. 

Figure  A-20  shows  on  log-log  plots  the  fitted  betas  from  measurements  over  an  81mm 
mortar.  The  left  plots  show  results  for  the  1st  time  gate  and  the  right  plots  are  for  the  4th 
time  gate.  The  top  two  plots  are  for  moderately  tilted  and  horizontal  81mm’s.  Similar  to 
the  60mm  plot  above,  there  are  some  outliers.  The  bottom  plots  are  for  81mm’s  oriented 
nearly  vertical,  nose  down.  In  this  orientation,  the  aluminum  tail  fin  of  the  mortar 
dominated  the  signal  and  resulted  in  very  different  values  for  the  polarizations.  Instead  of 
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the  ferrous,  cylindrical  solution  found  for  most  UXO  ( fix  >  (3V  =  fiz ),  it  gave  a  ferrous 
plate-like  solution  (fix  =  pv  >  /?. ).  The  “logarithmic”  circles  in  Figure  4  represent  the 

final  set  of  beta  polarizations  used  to  identify  the  buried  UXO  at  APG.  Each  circle  is 
centered  on  the  average  value  obtained  for  the  item  from  test  stand  and  APG  Calibration 
Grid  measurements.  The  “log”  circle  diameter  is  of  arbitrary  size,  but  does  a  reasonable 
job  in  capturing  most  of  the  betas  for  a  given  object.  The  circles  are  identified  in  the 
lower,  right  plot.  The  dashed  circle  represents  the  results  from  aluminum  tail  fins.  These 
“fin”  results  were  obtained  for  both  the  81mm  mortars  and  the  105mm  HEAT.  There  was 
some  scatter  in  results  for  the  larger  ordnance  between  nose  up  and  nose  down 
orientations,  but  the  UXO  library  set  plotted  was  adequate  to  capture  most  of  this  data.  It 
was  found  that  only  a  single  set  was  needed  for  the  105mm  (any  orientation)  and  105mm 
HEAT  (horizontal).  This  is  probably  an  indication  that  the  early  time  gates  of  the  EM61- 
HH  are  not  very  good  for  distinguishing  between  large  ordnance. 
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Figure  A-20  Fitted  parameters  for  an81mm  mortar  in  different  orientations 


Based  on  these  test  stand  results,  the  final  strategy  for  the  SAINT  at  the  APG 
demonstration  was  to  take  four  sets  of  data  over  each  item.  A  set  of  diagnostic  plots  was 
developed  to  check  on  both  the  data  and  fit  quality.  Samples  of  these  plots  are  shown  in 
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Appendix  A.  The  data  was  fit  unconstrained  in  beta  polarization  values  for  the  four  time 
gates  and  then  re-fit  with  the  library  of  expected  UXO  values  for  all  4  gates.  The  best 
library  fit  was  kept.  To  identify  UXO  from  clutter,  a  library  ratio  test  was  used.  The  ratio 
of  the  library  to  unconstrained  fit  error  was  calculated.  If  the  ratio  approached  1 .0,  both 
fits  were  equally  good  and  the  item  was  likely  the  identified  item.  If  the  ratio  was  small 
(by  observation  <  0.98),  the  item  could  not  be  fit  well  by  the  UXO  library  and  was  most 
likely  clutter. 

Test  measurements  were  also  taken  before  the  demonstration  with  items  placed  in  a  hole 
in  the  ground.  Measurements  at  the  test  site  found  a  large  gradient  in  the  EMI  signal  from 
the  ground.  The  1st  time  gate  of  the  EM61-HH  varied  over  100  mV  within  0.10  m  from 
the  ground  surface.  It  was  found  that  the  SAINT  system  could  be  used  to  make  a  quick 
vertical  profile  of  the  sensor  ground  response.  It  could  track  the  sensor  coil  height  as  it 
was  raised  and  lowered  from  the  surface  over  an  area  with  no  buried  objects.  Figure  A-21 
plots  this  ground  response  data  at  the  test  site  (black  curve)  and  at  the  Aberdeen  Proving 
site  (red  curve).  If  there  is  a  large  gradient  as  the  sensor  is  swept  about  over  the  ground, 
small  changes  in  height  above  ground  could  produce  noise  in  the  signal.  Because  of  the 
test  site  gradient,  the  sensor  had  to  be  swept  about  at  a  height  of  0.15  to  0.2  m  to 
minimize  this  noise.  At  APG,  there  was  no  measurable  signal  from  the  ground.  The 
sensor  was  swept  about  as  close  to  the  surface  as  was  possible. 


Figure  A-21  Comparison  of  sensor  response  as  a  function  of  height  at  two  sitess 
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APPENDIX  B  -  DATA  DIAGNOSTIC  PLOTS 
FOR  EM61  HH  &  SAINT  POSITIONING 
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APPENDIX  C  -  DATA  DIAGNOSTIC  PLOTS 
FOR  EM61  HH  &  TEMPLATE  POSITIONING 
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APPENDIX  D  -  APG  SCORING  RESULTS 

SAIC  (July  2009) 

TABLE  6a.  BLIND  GRID  TEST  AREA  RESULTS  (Template  -  Normal  Scoring) 


Response  Stage 

Discrimination  Stage 
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By  Depthb 

All  Mass 

Oto 

0.25  kg 

>0.25  to 

1  kg 

>1  to 
8kg 

All  Mass 

Oto 

0.25  kg 

>0.25  to 

1  kg 

>1  to 
8kg 

All  Depth 

1.00 

1.00 

0.98 

1.00 

1.00 

1.00 

0.35 

0.29 

0.24 

0.12 

0.42 

0.60 

0  to  0.15  m 

1.00 

1.00 

1.00 

1.00 

0.30 

0.13 

0.47 

0.50 

0.15  to  03  m 

1.00 

1.00 

1.00 

1.00 

0.25 

0.00 

0.14 

0.75 

0.3  to  0.6  m 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Background  Alarm  Rates 

Pbares:  0.42  |  Pbadisc:  0.14 

aThe  two  numbers  to  the  right  of  the  all  types  munitions  result  are  an  upper  and  lower  90- 
percent  confidence  interval  for  an  assumed  binomial  distribution. 

All  depths  are  measured  to  the  center  of  the  object. 


TABLE  7a.  BLIND  GRID  EFFICIENCY  AND  REJECTION  RATES  (Template  - 

Normal  Scoring) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

0.92 

0.71 

0.66 

With  No  Loss  of  Pd 

1.00 

0.31 

0.60 
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TABLE  8a.  BLIND  GRID  CORRECT  TYPE 
CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS  (Template  -  Normal  Scoring) 


Size 

Percentage  Correct 

25/37mm 

93% 

60/8 1mm 

87% 

105mm 

73% 

Overall 

84% 

TABLE  9a.  BLIND  GRID  MEAN  LOCATION  ERROR 
AND  STANDARD  DEVIATION  (Template  -  Normal  Scoring) 


Mean 

Standard  Deviation 

Northing 

— 

— 

Easting 

- 

— 

Depth 

0.095 

0.039 
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TABLE  6a.  BLIND  GRID  TEST  AREA  RESULTS  (Saint  -  Normal  Scoring) 


Response  Stage 

Discrimination  Stage 

Munitions*1 

Scores 

P”:  byty. 

ve 

P,fix:  by  type 

All 

Types 

105-mm 

81/60- 

mm 

37/25- 

mm 

All 

Types 

105-mm 

81/60- 

mm 

37/25- 

mm 

0.99 

0.97 

0.93 

0.98 

0.93 

0.83 

1.00 

0.97 

0.88 

1.00 

1.00 

0.93 

0.99 

0.97 

0.93 

0.98 

0.93 

0.83 

1.00 

0.97 

0.88 

1.00 

1.00 

0.93 

By  Depthb 

0  to  4D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

4D  to  8D 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

8D  to  12D 

0.67 

0.67 

0.00 

1.00 

0.67 

0.67 

0.00 

1.00 

Clutter 

Scores 

P cd 

Pfe 

By  Mass 

By  Depthb 

All  Mass 

Oto 

0.25  kg 

>0.25  to 
1kg 

>1  to 
8kg 

All  Mass 

Oto 

0.25  kg 

>0.25  to 
1kg 

>1  to 
8kg 

All  Depth 

1.00 

1.00 

0.98 

1.00 

1.00 

1.00 

0.43 

0.37 

0.31 

0.16 

0.52 

0.80 

0  to  0.15  m 

1.00 

1.00 

1.00 

1.00 

0.35 

0.13 

0.53 

0.83 

0.15  to  03  m 

1.00 

1.00 

1.00 

1.00 

0.50 

0.40 

0.43 

0.75 

0.3  to  0.6  m 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Background  Alarm  Rates 

Pbares:  0.42  |  Pbadisc:  0.16 

aThe  two  numbers  to  the  right  of  the  all  types  munitions  result  are  an  upper  and  lower  90- 
percent  confidence  interval  for  an  assumed  binomial  distribution. 
bAll  depths  are  measured  to  the  center  of  the  object. 


TABLE  7a.  BLIND  GRID  EFFICIENCY  AND  REJECTION  RATES  (Saint  - 

Normal  Scoring) 


Efficiency  (E) 

False  Positive 
Rejection  Rate 

Background  Alarm 
Rejection  Rate 

At  Operating  Point 

1.00 

0.63 

0.63 

With  No  Loss  of  Pd 

1.00 

0.63 

0.63 
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TABLE  8a.  BLIND  GRID  CORRECT  TYPE 
CLASSIFICATION  OF  TARGETS 
CORRECTLY  DISCRIMINATED 
AS  MUNITIONS  (Saint  -  Normal  Scoring) 


Size 

Percentage  Correct 

25/37mm 

100% 

60/8 1mm 

97% 

105mm 

77% 

Overall 

91% 

TABLE  9a.  BLIND  GRID  MEAN  LOCATION  ERROR 
AND  STANDARD  DEVIATION  (Saint  -  Normal  Scoring) 


Mean 

Standard  Deviation 

Northing 

- 

— 

Easting 

— 

— 

Depth 

0.122 

0.060 
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Appendix  E:  Points  of  Contact 


POINT  OF 
CONTACT 

ORGANIZATION 

Phone 

Fax 

e-mail 

Role  in  Project 

Dr.  Herb  Nelson 

ESTCP  Program  Office 
Arlington,  VA  22203 

702-767-3686  (V) 
202-215-4844  (C) 
herbert.nelson@osd.mil 

Program  Manager, 
MM 

Ms.  Katherine  Kaye 

HydroGeoLogic,  Inc. 
Reston,  VA  20190 

410-884-4447  (V) 
kkaye@hgl.com 

Program  Manager 
Assistant,  MM 

Dr.  Dean  Keiswetter 

SAIC 

Cary,  NC  27513 

919-677-1560  (V) 
919-454-3212  (C) 
dean.a.keiswetter@saic.com 

Principal 

Investigator 

Mr.  Glenn  Harbaugh 

Army  Research  Lab 
Welcome,  MD  20693 

804-761-5904  (V) 
roo749@yahoo.com 

Site  Safety  Officer 

Dr.  Bruce  Barrow 

SAIC 

Arlington,  VA  22202 

703-414-3884  (V) 
bruce.j  .barrow@saic.com 

Data  Analysis 

Mr.  Rick  Fling 

Aberdeen  Test  Center 

410-278-2999  (V) 
301-992-9080  (C) 
rick.  fling@us .  army.mil 

Test  Site  Manager 

75 


